83210 v1 The World Bank Group and Public Procurement A N I N D E P E N D E N T E VA L U AT I O N OCTOBER 2013 The World Bank and Public Procurement—An Independent Evaluation Volume I: Building Procurement Capacity and Systems © 2014 Independent Evaluation Group The World Bank Group 1818 H Street NW, Washington DC 20433 Telephone: 202-458-4497; Internet: www.ieg.worldbankgroup.org E-mail: ieg@worldbank.org Some rights reserved 1 2 3 4 17 16 15 14 This work is a product of the staff of the Independent Evaluation Group (IEG). Note that IEG and the World Bank does not necessarily own each component of the content included in the work. IEG and the World Bank therefore do not warrant that the use of the content contained in the work will not infringe on the rights of third parties. The risk of claims resulting from such infringement rests solely with you. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of IEG, the World Bank, its Board of Executive Directors, or the governments they represent. IEG and the World Bank Group do not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of IEG and the World Bank Group concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of IEG and the World Bank Group, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 Unported license (CC BY 3.0) http://creativecommons.org/licenses/by/3.0. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: IEG (Independent Evaluation Group). 2014. The World Bank and Public Procurement—An Independent Evaluation. Volume I: Building Procurement Capacity and Systems. Washington, DC: World Bank. License: Creative Commons Attribution CC BY 3.0 Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by the Independent Evaluation Group or the World Bank Group and should not be considered an official World Bank Group translation. IEG and the World Bank Group shall not be liable for any content or error in this translation. All queries on rights and licenses should be addressed to the IEG, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-3125; e-mail: ieg@worldbank.org. ISBN-13: 978-1-60244-236-8 ISBN-10: 1-60244-236-3 i Contents ABBREVIATIONS ................................................................................................................................ VII  ACKNOWLEDGMENTS........................................................................................................................ IX  1.  EVALUATION CONTEXT AND METHODOLOGY .................................................................... 1  Principal Areas of Focus ........................................................................................................................................... 3  Volume I—Building Client Country Procurement Capacity ................................................................................................... 3  Volume II —Contributing to Development Effectiveness in Bank-Financed Investment Projects......................................... 4  Evaluation Scope and Questions .............................................................................................................................. 5  Data Sources, Sampling Strategy, and Analytical Methods ...................................................................................... 7  Data Sources ......................................................................................................................................................................... 7  Sample Construction ............................................................................................................................................................. 8  Analytical Methods ................................................................................................................................................................ 9  2.  SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES ........................... 10  Country Strategies—Portfolio Review ..................................................................................................................... 10  CASs - Nature of References to Procurement—Diagnostics and Lending ................................................................ 11  CASs Areas of Procurement Discussed—Legislation, Institutions, Markets, and Integrity ........................................ 13  CAS Completion Reports—Assessment of Results Achieved ................................................................................... 15  CAS and CASCR Analysis—Trends over Time ......................................................................................................... 16  Country Strategies —Evidence from Country Case Studies ............................................................................................... 16  Country Resources Assigned to Procurement: Lending and Capacity Building ........................................................ 19  3.  SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS ............. 21  CPAR, MAPS, and PEFAs—An Introduction .......................................................................................................... 23  CPARs versus MAPS .......................................................................................................................................................... 24  Public Sector Management and the PEFA Instrument ........................................................................................................ 28  IEG Evaluation of Procurement Diagnostics—Methodology and Approach ........................................................... 30  Stakeholder Participation in ESW Preparation .................................................................................................................... 30  Client Government Participation ................................................................................................................................. 31  Donor Collaboration .................................................................................................................................................... 32  Civil Society, Academia, and Media Involvement ....................................................................................................... 32  Quality of the Bank’s Procurement Diagnostics (2002–12)..................................................................................... 32  Clarity of Objectives, Analysis, and Structure ..................................................................................................................... 32  Quality of Action Plan ................................................................................................................................................. 34  Use of Empirical Data ................................................................................................................................................. 36  Risk Assessment ........................................................................................................................................................ 36  Legal and Regulatory Framework .............................................................................................................................. 37  Institutional Framework............................................................................................................................................... 39  Capacity Levels and Capacity Development .............................................................................................................. 40  Fraud and Corruption.................................................................................................................................................. 40  Public Procurement Transactions, Complaints, Contract Administration, and Audits ................................................ 41  E-Procurement, Subnational Issues, and Green Procurement .................................................................................. 43  Procurement-Related NLTAs ..................................................................................................................................... 43  ii Quality of Results Achieved Through the Bank’s Procurement Diagnostics ........................................................... 44  Pillar I—The Legal and Regulatory Framework................................................................................................................... 48  Enhancing Open Competition ..................................................................................................................................... 49  Pillar II—Institutional Framework and Management Capacity............................................................................................. 50  Integration of Procurement into the Overall Public Financial Management System .................................................. 50  Establishing Functioning Procurement Authorities ..................................................................................................... 51  Human Capacity Development ................................................................................................................................... 52  Effective Information Systems Not Established .......................................................................................................... 53  Pillar III—Efficiency of Procurement Operations and Market Practices .............................................................................. 54  Pillar IV—Integrity and Transparency of the Public Procurement System .......................................................................... 57  Analysis of Specific Results in Public Sector Management—The PEFA Lens ....................................................... 60  Trends in Procurement Performance—Results Based on PEFA ........................................................................................ 60  Pairwise Comparison of Individual PEFA Scores ....................................................................................................... 60  Grouping Patterns of Change—Formal and Functional ............................................................................................. 61  Building Integrated Capacity for Procurement and Public Sector Management ................................................................. 63  Influence on Country Policies toward Public Financial Management and Procurement ............................................ 63  Influence on Country Integration of Procurement and Public Financial Management ............................................... 64  Influence on Stakeholders Outside the Public Sector ................................................................................................ 65  Evidence from Country Case Studies ..................................................................................................................... 66  Views in the Field – Quality of Diagnostic Instruments ....................................................................................................... 66  Views from the Field: Results .............................................................................................................................................. 69  The Bank’s Advisory Work—Results in Legal and Regulatory Reform ...................................................................... 69  Views from the Field: Human Capacity Development ......................................................................................................... 71  4.  SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING .......................... 74  Lending Operations That Supported Building Procurement Capacity..................................................................... 74  Characteristics of Bank Lending for Enhanced Procurement Capacity by Instrument........................................................ 81  Characteristics of Bank Lending for Enhanced Procurement Capacity—IEG Sample ....................................................... 86  Aggregating Scores at the Country Level ................................................................................................................... 87  Aggregating Scores at the Project Level .................................................................................................................... 87  Results of Efforts to Support Procurement Through Lending Operations ........................................................................... 88  Evidence from Country Case Studies.................................................................................................................................. 90  Targeted Lending for Procurement Reform—Mutually Reinforcing Loans ................................................................. 90  Targeted Lending for Procurement Reform—Slow Pace and Mixed Results ............................................................ 90  Targeted Lending for Procurement Reform—Missed Opportunities? ........................................................................ 91  Support to Project Implementing Units During Project Implementation...................................................................... 92  5.  INTEGRATING THE BANK’S PROCUREMENT SUPPORT .................................................. 94  Collaboration: Efforts and Outcomes ...................................................................................................................... 95  PREM, OPCS, and WBI Working in an Integrated Manner ........................................................................................ 95  Examples Where Coordination Could Be More Effective ........................................................................................... 96  Possible Ways Forward .............................................................................................................................................. 97  Supporting Country Procurement Capacity Through the World Bank Institute ................................................................... 97  World Bank Institute Instrument Choice ..................................................................................................................... 98  Outputs and Outcomes Achieved—Contract Monitoring ............................................................................................ 98  Outputs and Outcomes Achieved—Political Economy Analysis ................................................................................ 99  Outputs and Outcomes Achieved—Training on E-Procurement .............................................................................. 100  Outputs and Outcomes Achieved—Public Expenditure Management and Other Areas.......................................... 100  Overall Conclusions ........................................................................................................................................................... 102  iii 6.  BUILDING GLOBAL GOOD PRACTICE IN PROCUREMENT.............................................. 103  Contributing to Global Good Practice—OECD-DAC and the Aid Agenda ............................................................ 104  The Procurement Roundtable Initiative—2003–05 ........................................................................................................... 104  OECD-DAC Collaboration Following the Paris Declaration of 2005 ................................................................................. 106  The Heads of Procurement Group ........................................................................................................................ 108  Country Surveys—Views of Country Staff and Clients on Procurement Harmonization .................................................. 112  BIBLIOGRAPHY ................................................................................................................................. 114  ENDNOTES......................................................................................................................................... 121  Boxes Box 1.1. Public Procurement, Project Performance, and Development Outcomes .................................................................. 2  Box 2.1. Peru: Procurement in the Country Partnership Strategy ......................................................................................... 18  Box 3.1. Previous IEG Findings on Procurement from IEG’s Fiduciary Diagnostics Evaluation ............................................ 23  Box 3.2. Country Procurement Assessment Reviews ............................................................................................................ 24  Box 3.3. Methodology for Assessing Country Procurement System ..................................................................................... 26  Box 3.4. Integration of CPAR and MAPS Methodologies ....................................................................................................... 27  Box 3.5. Building Capacity Through Use of the PEFA Instrument .......................................................................................... 29  Box 3.6. Self-Assessments .................................................................................................................................................... 31  Box 3.7. CPARs and MAPS—Strengthening Clarity of Objectives and Structure .................................................................. 34  Box 3.8. Risk Assessments in Diagnostic Work—MAPS, CPARs, and Integrated Reports ................................................... 37  Box 3.9. Procurement Support to Turkey—Initial Positive Results of Advisory Work ............................................................. 48  Box 3.10. Achievements—Competitive Tendering ................................................................................................................. 50  Box 3.11. Integrating Procurement into the Budget Process .................................................................................................. 51  Box 3.12. Public Procurement Authority in Rwanda ............................................................................................................... 52  Box 3.13. Mauritius—Country Capacity and Good Governance for Effective Procurement Reform ...................................... 53  Box 3.14. Building Integrated Capacity—Public Sector Management and Procurement ...................................................... 64  Box 3.15. CPARs: Roadmaps for Procurement Reform—Especially Legal and Institutional ............................................... 66  Box 3.16. Early MAPS Exercises: Deemed to be Less Useful ............................................................................................. 68  Box 3.17. Results of Advisory Work—Legal and Regulatory Reform: Evidence from Field Visits......................................... 69  Box 3.18. Results—Building Human Capacity ....................................................................................................................... 72  Box 4.1. Supporting Procurement Capacity through PRSC Series ........................................................................................ 76  Box 4.2. Investment Lending/Technical Assistance Projects with a Procurement Focus ....................................................... 77  Box 4.3. Institutional Development Funds Support for Procurement Reform ......................................................................... 78  Box 4.4. Supporting Government Procurement Capacity Through Combinations of Programmatic DPLs ............................ 82  Box 4.5. Training Programs for Procurement—Training the Trainers in Bangladesh ............................................................. 84  Box 4.6. Investment Lending/Technical Assistance Projects—Afghanistan and Panama...................................................... 84  Box 4.7. Building Capacity Through Nontargeted Investment Lending .................................................................................. 92  Box 5.1. Developing Instruments and Markets to Promote Open Contracting ....................................................................... 99  Box 5.2. Performance Reform in Moldova—Using BOOST to Monitor Public Expenditures ................................................ 101  Box 6.1. Strengthening Procurement Through the Roundtable Working Groups ................................................................. 105  Box 6.2. Paris Declaration Monitoring Indicators on Procurement ....................................................................................... 106  Box 6.3. Harmonization of Procurement Guidelines ............................................................................................................. 110  Tables Table 2.1. Procurement Diagnostics Referred to in Country Strategies ................................................................................. 12 iv Table 2.2. Bank Instruments for Supporting Procurement Reforms ....................................................................................... 12 Table 2.3. Procurement in CASs—Outcomes and Indicators ................................................................................................. 13 Table 2.4. Procurement Issues in CAS Reports ..................................................................................................................... 13 Table 2.5. Procurement Issues in CAS Reports—Legislative or Regulatory Framework ....................................................... 13 Table 2.6. Procurement Issues in CAS Reports—Procurement Institutions and Country Capacity ....................................... 14 Table 2.7. Procurement Issues in CAS Reports—References to Integrity and Transparency .............................................. 15 Table 2.8. CAS Completion Reports—Results of Procurement Reforms ............................................................................... 15 Table 2.9. Strengthening of Country Procurement Systems as an Element of Past and/or Present CASs ............................ 17 Table 2.10. Project-Related Procurement Costs by Region, FY11 ($) ................................................................................... 19 Table 2.11. World Bank Procurement AAA Costs by Region and Year, FY02–11 ................................................................. 19 Table 3.1. Procurement Diagnostic Products, FY02–12 ......................................................................................................... 28 Table 3.2. ESW Reviewed by Product Type, Region, and FY ................................................................................................ 30 Table 3.3. Case Study Evidence: Country and Partner Ownership and Participation in ESW ............................................... 31 Table 3.4. Donor Collaboration by Region ............................................................................................................................. 32 Table 3.5. Overall Report Quality ............................................................................................................................................ 33 Table 3.6. Action Plan Quality by Time Period and Report Type............................................................................................ 35 Table 3.7. Integration of Procurement into the Budget Process and Procurement Monitoring ............................................... 39 Table 3.8. Transaction Efficiency, Complaints, Contract Administration, and Audits.............................................................. 42 Table 3.9. IEG Analysis of Results—Fiscal Years of CPARs/MAPS Reviewed ..................................................................... 45 Table 3.10. Best and Worst Scores by Dimension ................................................................................................................. 62 Table 4.1. Number of Procurement Projects, 2002–11 .......................................................................................................... 80 Table 4.2. Procurement Portfolio Profile, FY02–11 (number of countries) ............................................................................. 81 Table 4.3. Procurement Sample ($ millions) ........................................................................................................................... 86 Figures Figure 3.1. Overall Levels of Influence of Bank ESW ............................................................................................................. 45 Figure 3.2. Overall Levels of Improvements in Each Dimension ............................................................................................ 46 Figure 3.3. Improved Procurement Reform Supported by Bank AAA ..................................................................................... 46 Figure 3.4. Substantial Impact on Legislative Reform—Less on Action Plans ...................................................................... 49 Figure 4.1. Operations That Focused on Building Procurement Capacity, 2002–11 .............................................................. 80 Appendixes are available at: https://ieg.worldbankgroup.org/evaluations/world-bank-group-and-public-procurement v Abbreviations AAA Analytical and advisory assistance ADB Asian Development Bank CAS Country Assistance Strategy CASCR Country Assistance Strategy Completion Report CDD Community-driven development CPAR Country Procurement Assessment Report CPI Compliance performance indicator DPL Development policy loan DPO Development policy operation ESW Economic and sector work ICB International competitive bidding ICR Implementation Completion Report IDA International Development Agency IDB Inter-American Development Bank IDF Institution development fund IEG Independent Evaluation Group IFI International financial institution ISR Implementation status report JICA Japan International Cooperation Agency MAPS Methodology for Assessing Procurement Systems MDB Multilateral development bank NLTA Non-lending technical assistance OECD Organisation for Economic Co- operation and Development- Development OECD-DAC Development Assistance Committee, Organisation for Economic Co-operation and Development OPCS Operations Policy and Country Services PEFA Public expenditure and financial accountability PER Public expenditure review PPP Public-private partnership PREM Poverty Reduction and Economic Management Vice Presidency/Network PRSC Poverty Reduction Support Credit WBI World Bank Institute vii Acknowledgments This report of the Independent Evaluation Group (IEG) was prepared by a core team led by Anjali Kumar, with major contributions provided by Nils Fostvedt, Prem Garg, Gita Gopal, Ian Hume, Eduardo Maldonado, Arvind Nair, Carla Pazce, Juliane Piecha, Robert Rothery, Swizen Rubbani, Justin Sigman, Kathryn Steingraber, Nadine Tushe, Christine Wallich, and Clay Wescott. Eleven country case studies were undertaken by Gian Casartelli, Jorge Claro, Ajay Guha, Rodolfo Hernandez, Jean-Jacques Lecat, Robert Rothery, Paul Schapper, and Christine Wallich. Additional contributions were provided by Jeanmarie Fath Meyer and S. Janakiram. Corky de Asis, Aimée Niane, Agnes Santos, and Lily Tsang provided administrative support. The team gratefully acknowledges the support of managers and staff throughout the World Bank, especially in Operations Policy and Country Services departments and from both former and current regional procurement managers. IEG appreciates the many individual procurement staff throughout the Bank who have been generous with their time for surveys and interviews on aspects of the Bank’s procurement system. Particular thanks are offered to the field-based procurement staff that provided assistance for field visits in Azerbaijan, Bangladesh, Ethiopia, Indonesia, Mexico, Morocco, the Philippines, Peru, Senegal, Tanzania, and Turkey, arranging meetings with key officials as well as private persons. IEG also extends its thanks to other areas of the Bank consulted during the course of this evaluation: the Controllers unit, Economic Management, Institutional Integrity, Internal Audit, and Legal units. IEG also appreciates exchanges with the African Development Bank, the Asian Development Bank, the European Bank for Reconstruction and Development, the European Commission, the European Investment Bank, the Inter-American Development Bank, and the Organisation for Economic Co-operation and Development, who shared their perspectives with the evaluation team, and its exchanges with academia at George Washington University, as well as Harvard University, the University of Florida and Nottingham University, UK. The team also thanks client government counterparts, private sector suppliers, project implementation units, personnel from implementing agencies, and members of civil society who participated in IEG’s surveys and interviews. The evaluation benefitted from constructive advice from external advisers and peer reviewers. Peer reviewers included Praful Patel (former Vice President, South Asia Region, and former Director, Infrastructure, Africa Region); Christiaan Poortman (former Vice President, Middle East and North Africa Region, World Bank, and ix ACKNOWLEDGEMENTS current Senior Advisor at Transparency International and Chair of the Board of the Construction Sector Transparency Initiative); and Dr. Peter Trepte (Senior Fellow in Public Procurement Law, University of Nottingham). An external advisor provided overall guidance to the study—Professor Daniel Gordon, Associate Dean for Government Procurement Law, at the George Washington University Law School. The evaluation was conducted under the guidance of Nicholas York, Director, and Ali Khadr, Senior Manager, IEG Country, Corporate, and Global Evaluation, and Caroline Heider, IEG Director General. x 1. Evaluation Context and Methodology Good public procurement practices are a major determinant of the effectiveness of public expenditure. On behalf of their citizens, governments typically spend as much as 5–20 percent of their gross domestic product on procurement of goods and services, and effective procurement policies enable better use of government budgets (OECD 2011b). Good national procurement practices are therefore an essential element of the poverty reduction focus of the Bank. Good procurement in Bank projects is also associated with better development outcomes (Box 1.1). Equally, sound public procurement in client countries is a prerequisite for the success of the Bank’s newly introduced Program for Results lending instrument. This instrument uses national procurement policies and may provide a vehicle to allow procurement practices in Bank lending to be unified across investment and policy-based lending, as well as harmonized with other donors.1 World Bank–financed investment lending operations, which dominate Bank lending, can also constitute a significant element of public procurement in client countries. The Bank’s procurement policies and processes affect the development impact of Bank lending and influence public procurement practices in client countries. The Bank seeks to ensure that its funds are used for the purpose intended and that they support development effectively and efficiently. Thus, the twin issues that underpin this Independent Evaluation Group (IEG) evaluation are first, how effectively has the World Bank helped build well-functioning public procurement systems in client countries (Volume I) and second, how well have Bank procurement policies and procedures for its investment lending supported the development effectiveness of Bank lending (Volume II)? The evaluation parallels an intensive review by Bank management of the institution’s procurement function, motivated by the need to respond to a range of internal and external changes in the Bank’s procurement environment.2 Management points toward changes in the relevance of the present system, given changes in country capacity, supplier patterns, and new global practice emerging in public procurement. IEG corroborates, for example, that in aggregate, contracts won by international suppliers (under the Bank’s international competitive bidding procurement method)—as a proportion of overall investment lending by value— have declined, from around 50 percent in FY00–02 to around 30 percent in 2011.3 In view of such changes, management proposes to add flexibilities to the present system, based on best-fit-for purpose, anchoring changes in terms of development 1 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY effectiveness and value for money for clients.4 This evaluation is intended to contribute to the formulation of specific proposals for change in the latter phases of the management review. Box 1.1. Public Procurement, Project Performance, and Development Outcomes To what extent does good procurement help achieve good lending outcomes? An IEG analysis of data from Implementation Completion Report (ICR) Reviews and from self-reporting by staff through Implementation Status Reports (ISRs) suggests that procurement performance is closely associated with project success. In projects with unresolved procurement problems at closing (that is, with procurement performance ratings of Moderately Unsatisfactory or worse in the last ISR), the likelihood of unsatisfactory outcomes is more than three times as high as for projects with satisfactory performance at completion. For projects not yet closed, the dependence appears stronger— unsatisfactory rating on procurement increases the likelihood of unsatisfactory development outcome ratings more than fivefold. Procurement Performance and Development Outcomes—Completed Projects Procurement ratings Outcome ratings MS+ MU- Total MS+ 340 14 354 MU- 107 37 144 Total 447 51 498 Percent of total MS+ 76 27 71 MU- 24 73 29 Total 100 100 100 Source: IEG analysis of World Bank data. Note: Data are complete to November 8, 2012. Procurement ratings are based on the latest ISR; outcome ratings are based on IEG's ICR Reviews. These data include IBRD/IDA funded investment lending only. Unresolved procurement problems are also associated with increased risks to development outcomes (that is, obtaining a risk to development outcome rating of “significant or higher” in IEG’s ICR Reviews), indicating uncertain sustainability for projects with problematic procurement. Procurement Performance and Risks to Development Outcomes—Completed Projects Procurement ratings Risk to development outcome MS+ MU- Total Moderate or lower 261 18 279 Significant or higher 162 30 192 Total 423 48 471 Percent of total Moderate or lower 62 38 59 Significant or higher 38 63 41 Total 100 100 100 Source: IEG analysis of World Bank data. Note: Data are complete to November 8, 2012. Procurement ratings are based on the latest ISR; outcome ratings are based on IEG's ICR Reviews. Data include IBRD/IDA funded investment lending only. 2 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY Management’s review, as well as questions raised by the Bank’s Board, have informed the scope of this IEG evaluation. IEG also covers issues regarding Bank procurement that have been raised by other stakeholders at the corporate, country, and sector level.5 References have been made to Bank efforts to build country capacity, but also to capacity constraints for both the Bank and client countries.6 The termination of the recent pilot on the use of country systems has been queried, as have Bank and donor harmonization in procurement (IEG 2011e). Several evaluations point to the value of coordination between procurement and public expenditure reforms, but also to the difficulties of realizing such coordination (for example, IEG 2008b, 2010e). A large number of evaluations point to client country corruption issues, whereas some also mention such issues in Bank projects. References to efficiency are among the most prominent, especially delays caused by procurement issues, often associated with an escalation of project cost. Bank regions allude to persistent process difficulties, system inefficiencies, and lags in execution. Finally, there are queries related to the adaptability of current Bank procurement guidelines to new contexts, such as public-private partnerships (PPPs); technology loans; and small, fragile, or conflict-affected states.7 These questions have also contributed to the evaluation’s design and coverage. Principal Areas of Focus VOLUME I—BUILDING CLIENT COUNTRY PROCUREMENT CAPACITY In this evaluation, strengthening national procurement systems refers to the full set of institutional arrangements—the rules, norms, and procedures, both formal and informal—governing public procurement, as well as the human resources organized to undertake public procurement transactions. Bank contributions to this process have taken the form of both advisory services and support through lending. Procurement diagnostics were supported first through the Bank’s Country Procurement Assessment Reports (CPARs) and later through the internationally agreed-on Methodology for Assessing Procurement Systems (MAPS). More integrated diagnostics of public procurement and public financial management systems were also attempted through the Public Expenditure and Financial Accountability (PEFA) instrument. Support for strengthening national procurement systems has also been offered through some Bank lending operations, notably through development policy operations (DPOs) that focus on fiscal management or public sector reform and that sometimes include procurement reform in their scope. The Bank had a limited number of investment loans with a significant focus on procurement. More 3 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY significant, though small in value, substantial use has been made of trust-funded institutional development fund (IDF) grants for procurement support. In addition, Bank investment lending, even if not focused on procurement reform per se, frequently provided procurement capacity-building support to project implementation units to encourage learning by doing. IEG’s review of Bank support for capacity development looks at the relevance and quality of support and the extent to which it has contributed to sustained results. IEG also evaluates the extent to which the Bank contributed to the convening of global fora and the formulation of global norms on procurement. VOLUME II —CONTRIBUTING TO DEVELOPMENT EFFECTIVENESS IN BANK-FINANCED INVESTMENT PROJECTS Policies and procedures on procurement under Bank-financed investment operations also seek to support development effectiveness in client countries by ensuring that funds are used as economically and efficiently as possible, for the purpose intended.8 Core principles in Bank Guidelines also refer to the encouragement of competition (equal opportunity for all bidders) for the development of domestic markets and transparency in the procurement process.9 The evaluation therefore looks at the effectiveness of current Bank procurement systems, in terms of the achievement of these underlying principles, including the extent to which there may be trade-offs in their achievement. It will also review current practice in those areas of Bank procurement where future changes are proposed, notably, in terms of the use of country systems—a theme that is integral to both Volume I and Volume II of this report. IEG evaluates the Bank’s 2008 pilot program on the use of country systems in procurement as well as present stakeholder views, to inform the Bank’s future direction in this regard (World Bank 2008b, 2010e). Bank lending has evolved in new directions, ones not envisaged by its present procurement framework. Recent years have seen the introduction of new delivery vehicles, and new areas of Bank support. These include, for example, an uptake in community-driven development (CDD) projects that pose particular questions, as they give control for procurement decisions to community groups, and increased Bank support to fragile and conflict-affected countries.10 PPPs, where the Bank contributes financing to projects not wholly under the control of client countries’ government but also to projects involving private financing, also raise questions in terms of procurement. And the Bank is providing more support in complex areas such as information technology and communications, where its traditional separation of lending and consultancy services is less distinct and where assumptions of buyers’ prior knowledge of the best systems is open to question. New global procurement practices may be better suited to some of these areas.11 4 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY Given the Bank’s fiduciary obligations and its heightened focus in recent years on fraud and corruption, a key element of the evaluation is the extent to which procurement risk, and hence overall fiduciary risk, is handled in Bank-financed projects. Its prior review of and multistage “no-objection” to a significant part of contracts have been a cornerstone of its risk-management framework. Procurement transactions subject to prior review have averaged 53 percent by value over the past decade.12 There is a perception among Bank staff that greater concern about corruption in procurement has resulted in more intensive scrutiny, leading to project delays; this suggests a trade-off between increased transparency and efficiency (IEG 2011d). Some aspects of the Bank’s management of procurement-related risk have been reviewed by the Bank’s Internal Audit Vice Presidency. IEG’s analysis of the Bank’s procurement risk-management framework takes note of the Vice Presidency’s existing analyses and reviews complementary aspects. Finally, IEG reviews Bank procurement processes, system efficiency, and the scope for future improvement not only through adjustments in Bank systems, but also through new procurement platforms and processes. Evaluation Scope and Questions Specific evaluation questions are spelled out here. Two overarching questions form the core of this evaluation: 1. To what extent has the Bank helped its clients develop better procurement capacity and improve their public procurement systems? 2. To what extent does the application of Bank procurement guidelines in its investment lending help support its own development effectiveness objectives, in terms of fostering economy, efficiency, and transparency in the execution of Bank projects in client countries? Questions specific to particular areas of focus linked to the overarching framework are as follows: 1. Building Client Country Procurement Capacity  To what extent have World Bank country departments supported the strengthening of client procurement systems through diagnostic and advisory work and non-lending technical assistance?  To what extent has the World Bank been able to provide support for procurement system reform through its development policy and investment lending platforms? 5 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY  Has the Bank been able to integrate advisory work in procurement within a framework of public financial management and efficient public expenditure? 2. Assisting Global Good Practices in Public Procurement  To what extent has the Bank been able to coordinate/harmonize its procurement processes with other development partners in accordance with the global aid effectiveness agenda, and to what extent has it been able to adopt useful elements from the experience of other multilateral development banks (MDBs) and international financial institutions (IFIs) in its own procurement systems? 3. Contributing to Development Effectiveness in Bank Lending  How effective is the Bank’s present investment lending procurement system, in terms of the extent to which users of the system, at the level of clients and task team leaders, perceive the costs and benefits of the present process?  To what extent has the Bank been able to move toward the use of country systems, as envisaged in the aid effectiveness agenda? To what extent could previous efforts in this direction have been strengthened?  To what extent are new approaches to procurement, such as overall value for money (including social and ecological impact), taken into account in the Bank’s procurement policies? To what extent is contract management incorporated in Bank procurement systems? 4. Adapting to New Areas of Lending and New Procurement Methods  To what extent has the Bank’s present procurement system accommodated the evolving needs of Bank lending (based on a review of procurement issues with regard to CDD projects, public-private partnerships, and information and technology projects)?  To what extent has the Bank’s present procurement system facilitated Bank engagement in fragile and conflict situations? 5. Managing Risk  Have the recent changes in the Bank’s fiduciary environment for procurement qualitatively improved its overall risk-management framework and enhanced its overall development effectiveness?  Have risk-mitigation measures (such as thresholds for prior review) been applied with consistency and efficiency? 6. Increasing Efficiency in Bank Lending 6 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY  How effective is the Bank’s procurement system in terms of the distribution and utilization of Bank procurement resources? In this volume, IEG addresses the first overarching evaluation question, that is, the extent to which the Bank has helped its clients develop better procurement capacity and improve their public procurement systems. Volume II addresses the second overarching question—the extent to which the Bank’s procurement guidelines and processes, in its investment lending, help support its own development effectiveness objectives. Data Sources, Sampling Strategy, and Analytical Methods DATA SOURCES IEG used a range of information sources in its evaluation:  IEG evaluative information. To formulate the areas of focus and questions addressed, IEG undertook a review of all major evaluations that included significant references to the Bank’s procurement function, including corporate, country, and sector evaluations (Appendix A) (IEG 2012b).  Eleven country case studies. Field visits were based on structured questions, addressed to different groups of interlocutors, to capture the full spectrum of perceptions on procurement, including Bank management, country procurement staff, task team leaders, government clients, private sector suppliers, and civil society organizations involved in procurement oversight.13  Procurement staff interviews, based on structured questionnaires, for specialized areas of procurement: Information and communications technology projects and fragile and conflict-affected states.  Additional, nonstandardized procurement staff and task team leader interviews, with open-ended responses.  Reviews of relevant Bank documents (Country Assistance Strategies [CASs] and their IEG Completion Reports [CASCRs]), lending-related documents (program and project appraisal documents, Implementation Status and Results Reports, and associated completion reports), and analytic and advisory work on procurement.  Analysis of procurement-related documents prepared by the Bank, including those related to the use of country systems pilot.  Analysis of procurement-specific Bank data, including the form 384 data set, regional procurement tracking systems (for example, PROCSYS in Africa), the Procurement Risk Assessment Management System database, and postreview information. 7 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY  In-depth review of specific contracts based on a data set constructed by IEG through its field visits, covering more than 500 contracts in these countries.  Analysis of data sets maintained by the Integrity Vice Presidency, including its complaints and cases.  Analysis of external evidence of the role of World Bank in global fora on procurement, and in relation to other donors based on documentation as well as interviews with relevant staff in other agencies, including the Development Assistance Committee of the Organisation for Economic Co- operation and Development (OECD-DAC), the European Union, and other IFIs and MDBs. SAMPLE CONSTRUCTION Although IEG undertook complete enumeration of all available reports in some areas (the review of CASs and CASCRs, the review of major IEG reports referring to procurement), other desk reviews covering a 10-year period (2002–11) and field visits were undertaken on a country sample basis. IEG constructed a purposive sample based on the following:  Inclusion of all Bank regions and both higher- and lower-income countries  A record of significant, varied, and identifiable capacity building activities in procurement, whether economic and sector work (ESW), lending that supported procurement, and/or IDF grants  A reasonable proportion of investment lending, with a diverse portfolio that illustrates different procurement processes; for example, large infrastructure projects, as well as decentralized delivery, as in health or education  The inclusion of countries that had participated in the Bank’s country systems pilot, as such countries underwent extensive diagnostic studies on country capacity, and also in view of the interest in the pilot.  The views of each regional procurement team, and relevant country teams regarding field-based visits.14 All Bank borrowers were mapped against these criteria in a “country selectivity framework” (Appendix Table A.6). This process led to the selection of a first list of 11 countries for field visits and another 10 countries for desk review, bringing the total to 21 countries.15 In some areas of the analysis, additional countries were included if data were already available (for example, in the CAS or PEFA analyses). IEG did not apply the country selectivity framework to specialized topics on procurement, such as PPPs, CDD projects, information and communication technology projects, or fragile and conflict-affected countries. IEG used focused project or country-based selection, as described in the relevant sections, to construct larger and more representative samples. 8 CHAPTER 1 EVALUATION CONTEXT AND METHODOLOGY ANALYTICAL METHODS IEG used both qualitative and quantitative methods. Structured questionnaires, for portfolio analysis and interviews, used category building and scoring to enable summarizing and comparison. Quantitative data were reviewed using spreadsheet and statistical analyses. Where data permitted, IEG undertook statistical correlation or regression analyses that associated results in specific areas with indicators such as country performance or country governance. In many areas, Bank procurement data were only partially available, or not available in a manner that could be extracted for analysis. Data across regions were often not comparable, sometimes making it difficult to extract Bank-wide messages. Some core parameters on procurement functions are not tracked. As a result, IEG had to construct its own, sometimes partial, data sets. Results of analysis are subject to this caveat. One methodological challenge was the measurement of effectiveness of Bank interventions for country procurement capacity development. IEG synthesized and triangulated information from time-ordered sequences of policy information from existing diagnostic or lending documentation (such as goals and objectives, advisory support, lending support, and efficiency parameters), and information from country case studies, including open-ended survey information. 9 2. Support for Procurement Reform in Country Strategies To what extent has procurement reform been a priority for country programs in the Bank? The implied subquestions are: Have country departments recognized the importance of procurement and provided adequate resources both for diagnostics and for the implementation of change? What aspects have been emphasized and what instruments favored? And to what extent are planned interventions translated into reality, and if so, have they led to successful outcomes? In the country-led model that dominates Bank resource allocation, the extent to which country units take ownership of and give support to any issue critically influences the extent to which it is reflected in the work program. Main Findings  IEG’s analysis shows that Bank management had a high level of awareness of procurement-related issues and the need for support to build client countries’ capacity. Issues relating to integrity and transparency were also frequently discussed, market development less so.  The Bank made efforts to provide support for procurement reform to many countries. However, there was a loose translation of priorities from procurement discussions in country strategies to specific actions in country work programs. Most attention, especially initially, focused on existing or planned diagnostics. The Bank’s CPARs were given a great deal of attention in CASs; other instruments received negligible attention.  In terms of resources, the bulk of Bank resources were devoted to project support for procurement, which was far higher than support for analytical or advisory work. Although not easy to trace, there appear to have been limited resources allocated for procurement capacity building.  There has been a marked decline over time in support to procurement-related analytical and advisory assistance (AAA) after 2007. Country Strategies—Portfolio Review The core exercise IEG used to review these questions was a structured analysis of all World Bank country strategy documents between 2002 and 2011, referred to in this 10 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES note collectively as CASs (Appendix Tables B.1 and B.2).16 IEG reviewed more than 260 strategies, covering 124 countries, including 72 IDA (International Development Association) countries. For a limited number of countries, two or more CASs were available, spanning a period of time over the decade.17 Next, to examine results achieved, IEG reviewed relevant CAS Progress Reports, CASCRs, and IEG’s CASCR Reviews. To the extent available, IEG also attempted to review annual budget process allocations to Bank fiduciary units in each region, to capture procurement- related work not charged to individual projects. Finally, IEG supplemented this information by drawing on information from interviews with country management, on their commitment to procurement reform, obtained from IEG case studies. CASs - Nature of References to Procurement—Diagnostics and Lending Over the period 2002–11 as a whole, procurement has been referred to in virtually all CASs; of these, a significant proportion allude to programs for strengthening procurement. Of the 267 country strategies reviewed, as many as 254 (95 percent) made some reference to procurement (Appendix Table B.3). Of these, 203 made reference to an existing or planned diagnostic; around 150 envisaged further support for procurement reform through investment lending, policy lending, or technical assistance; and around 115 included procurement outcomes or monitoring indicators built in to the CAS. References to difficulties encountered from procurement bottlenecks were also encountered. However, although more than 180 reports mention procurement- related difficulties, such references were mainly to their impact on country development (169 reports).18 Although reference was also made to procurement factors that impacted the Bank’s lending program, that number was considerably smaller (58 reports). Procurement diagnostics were referred to frequently—primarily the Bank’s CPARs, which were mandatory until 2006. The strategy documents of 95 countries (reflected in a total of 142 reports) mentioned the existence of a previously undertaken procurement diagnostic. There were 128 references to a CPAR, in 89 countries, though only 4 references to procurement diagnostics as part of a PEFA, and only 1 reference (Sierra Leone 2010 CAS)19 to a MAPS diagnostic (Table 2.1).20 Given that the number of CPARs tapered off considerably after 2007, these data illustrate that during the period of mandatory CPARs, there was more systematic attention to the issue of procurement in country strategies. 11 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Table 2.1. Procurement Diagnostics Referred to in Country Strategies Procurement diagnostic Number of reports Number of countries CPAR 128 89 PEFA 4 4 MAPS 1 1 Total 142 102 Source: IEG analysis of CASs. Note: Covering 102 countries, as different diagnostics may have been undertaken at different times for the same country. CPAR = Country Procurement Assessment Report; MAPS = Methodology for Assessing Procurement Systems; PEFA = Public Expenditure and Financial Accountability Assessment. Once diagnostics were undertaken, they usually led to an articulated procurement strategy, though a large element of follow-up support (87 countries) included further diagnostic (Table 2.2). The Bank also offered follow-up support through technical assistance in 79 countries (largely through IDF grants).21 Support through development policy loans (DPLs) also occurred frequently (55 countries), though these were typically broad-based operations where procurement is one among several topics addressed.22 In contrast, there was modest planned support through investment lending—in only 13 countries. These findings correspond well with data on Bank operations involving procurement. Table 2.2. Bank Instruments for Supporting Procurement Reforms Instrument Number of reports Number of countries Further diagnostic work 120 87 Support through technical assistance/grantsa 115 79 Support through DPL 74 55 Support through investment lending 14 13 Source: IEG analysis of CASs. a. Technical assistance/grants refers mainly IDFs but also includes programs like the Governance Partnership Facility and other grant- making programs, although sometimes not specifically mentioned. It also refers to the Bank’s technical assistance loans/grants. DPL = development policy lending. To the extent that the Bank had planned outcomes for support to procurement, it was usually reflected in related CAS indicators. Eighty-six reports (for 66 countries) included procurement reform as one of the CAS outcomes, and 79 (from 61 countries) also had procurement-related monitoring indicators. In another 30 reports there were procurement-related monitoring indicators, even though there were not any CAS outcomes for procurement (Table 2.3). This underlines the somewhat loose connection between country program statements and procurement reform undertaken in practice, including the measuring and monitoring of procurement reform. 12 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Table 2.3. Procurement in CASs—Outcomes and Indicators Procurement-related CAS outcomes No Yes Total Procurement related monitoring No 151 7 158 indicators Yes 30 79 109 Total 181 86 267 Source: IEG analysis of CASs. Note: CAS = Country Assistance Strategy. CASs Areas of Procurement Discussed—Legislation, Institutions, Markets, and Integrity IEG grouped procurement-related issues into four broad sets: legislative and regulatory areas, procurement institutions and country capacity, efficiency of procurement operations and development of markets, and issues related to integrity/transparency and fraud and corruption (Table 2.4).23 Table 2.4. Procurement Issues in CAS Reports Number of Number of Procurement issue mentioned in report reports countries Are the key issues identified in the areas of the legislative or regulatory 68 54 framework? Are the key issues identified in the areas of procurement institutions and country 121 85 capacity? Are the key issues identified in the areas of efficiency of procurement operations 74 59 and markets? Are the key issues identified in the areas of Integrity and transparency/fraud and 106 74 corruption? Source: IEG analysis of CASs. Procurement issues in the first area, the legislative and regulatory framework, were mentioned to a lesser extent than other issues—68 reports for 54 countries (Table 2.5).24 Of these, around half (32) describe Bank support for procurement reform in this area (Appendix Table B.6). Of the 26 country strategy reports for which subsequent documents were available, related improvements were indicated for 20, with a reasonable degree of progress but clear scope for improvement. Table 2.5. Procurement Issues in CAS Reports—Legislative or Regulatory Framework Number of Issue Number of reports countries CAS reports identifying key issues in the areas of: the legislative or 68 54 regulatory framework? of which: CAS reports mentioning Bank support for the legislative and 32 28 regulatory framework? CAS completion reports mention some improvement in the legislative or 30 27 regulatory framework? Source: IEG analysis of CASs. Note: CAS = Country Assistance Strategy. 13 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Issues in the second area, institutional framework and staff/management capacity, were raised most frequently, typically with reference to the lack of and/or ineffectiveness of institutions in the country to undertake or oversee public procurement (Table 2.6). There were also references to the lack of qualified procurement specialists in the country. However, follow-up Bank support for the strengthening of the institutional framework was not always planned—again, there is reference to follow-up in around half. Of the 121 CASs that mentioned issues relating to procurement institutions and country capacity, 69 alluded to Bank support; somewhat less than two-thirds (40) mention resulting improvements.25 Bank follow-up therefore appears to have been sporadic, though at least partially successful once undertaken. These numbers could also reflect the difficult and slow-moving nature of achieving improvements in domestic institutional capacity. Table 2.6. Procurement Issues in CAS Reports—Procurement Institutions and Country Capacity Number of Number of Issue reports countries CAS reports identifying key issues in the areas of procurement institutions 121 85 and country capacity of which: CAS reports mentioning Bank support for institutional framework and 69 55 staff/management capacity/dissemination of information CAS completion reports mentioning some improvement in the institutional 40 32 framework Source: IEG analysis of CASs. Note: CAS = Country Assistance Strategy. Issues related to the third area, efficiency of procurement operations and markets, were addressed less frequently. Such issues include process improvements to improve delays, reduce costs, or undertake procurement more efficiently, for example, through the development of e-procurement systems to enhance efficiency. Fifty-nine countries had strategy reports that mentioned issues related to procurement efficiency; 20 mentioned Bank support for reform in this area, and of these, 9 mentioned improvement the completion report (Appendix Table B.7). In the fourth area, mention of issues relating to integrity and transparency/fraud and corruption were almost as frequent as mentions of country capacity. Procurement issues relating to integrity and transparency were mentioned in 106 country strategy reports; 39 of these also included statements of Bank support for procurement reform in this area.26A surprisingly high 35 CASCRs mentioned an improvement in integrity and transparency/fraud and corruption (Table 2.7). 14 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Table 2.7. Procurement Issues in CAS Reports—References to Integrity and Transparency Number Number of Issue of reports countries CAS reports identifying key issues in the areas of: integrity and 106 74 transparency/fraud and corruption in procurement of which CAS reports mentioning Bank support in this area 39 36 CAS completion reports or progress reports mention some improvement in 35 30 integrity and transparency/fraud and corruption Source: IEG analysis of CASs. Note: CAS = Country Assistance Strategy. CAS Completion Reports—Assessment of Results Achieved To what extent do CASs provide evidence of success achieved by the Bank in supporting capacity building in procurement? IEG reviewed 167 strategy documents from 102 countries for which CASCRs, CAS progress reports, and/or CASCR reviews were available over this period, scoring reports based on the extent to which results were actually achieved (Table 2.8). Table 2.8. CAS Completion Reports—Results of Procurement Reforms Nature of support 0 % 1 % 2 % 3 % Total Legislative and regulatory framework 85 51 6 4 43 26 33 20 167 Institutional framework and staff/management 96 57 7 4 39 23 25 15 167 capacity/dissemination of information Efficiency of procurement operations and markets 127 76 7 4 24 14 9 5 167 (better delegation/community driven projects); support for PPP; e procurement; credit/contracting processes Integrity and transparency of the public procurement 112 67 13 8 33 20 9 5 167 system (disclosure, complaints mechanism, dispute resolution, external audit) Harmonization and UCS (harmonization with other 137 82 8 5 19 11 3 2 167 donors, with country systems) Source: IEG analysis of CASs. Note: Scores assigned 3 = results were achieved to a great extent; 2 = results were achieved to some extent; 1 = very little extent; 0 = no reference to procurement or no mention of improvement in the area in which procurement reform had been attempted/undertaken. CAS = Country Assistance Strategy; PPP = public-private partnership; UCS = use of country systems. Results show that the Bank had modest success in achieving at least some progress in the areas of legislative and institutional issues (49 percent and 43 percent, respectively), though poor results in “second-generation” aspects of procurement reform: market efficiency (24 percent); the promotion of integrity and transparency (33 percent); and harmonization of use of country systems (18 percent). 15 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES CAS and CASCR Analysis—Trends over Time IEG’s analysis also explored the extent to which there were any significant shifts over time in responses to some key questions addressed during the evaluation period. IEG constructed moving three-year averages from 2004 (first three years) to 2011.27 Results do indicate some shifts (Appendix Table B.13). CAS discussions tended to become more remote from recent diagnostic reports, suggesting that the stock of reports that were already undertaken could have become less current. There was also some decline in the focus on issues in legislative and regulatory frameworks, perhaps again suggesting diminishing CPARs, but perhaps also that gradually countries moved toward a “second generation” of reform issues. Meanwhile, allusions to institutional strengthening and integrity related issues increased. In terms of the content of planned follow-up to CAS discussions, there was a decline in the link between actions proposed and earlier diagnostic work and also a decline in further diagnostic work as a vehicle of support. Support through DPLs and technical assistance/grants remained steady, and support through investment lending remained at modest levels. In terms of content, there are indications of a movement away from support for legislative and regulatory reform and toward efforts to build institutions. Bank support for efficiency/markets has remained low, albeit with some increase. Support for the use of country systems declined, perhaps reflecting the stalling of reform efforts in this area. Finally, IEG findings suggest a stable degree of modest progress, based on CASCRs/CAS progress reports (Appendix Table B.14). COUNTRY STRATEGIES —EVIDENCE FROM COUNTRY CASE STUDIES Evidence from IEG’s country case studies largely corroborates the findings from the portfolio review (Table 2.9). Improving country procurement capacity was reflected in country strategies to a substantial extent in most countries—a view shared by country management and Bank procurement staff. However, the translation of intentions into the work program was less frequent, and few countries systematically tracked the achievement of procurement-related objectives over time. Yet field visits also illustrated that in some countries, CAS focus was substantial, albeit with limited follow-up, as well as cases where CAS focus was modest, but there was strong work program engagement nevertheless. 16 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Table 2.9. Strengthening of Country Procurement Systems as an Element of Past and/or Present CASs Bank Country procurement Question Average management staff To what extent has procurement capacity development been an 3.2 3.1 3.2 objective of the CMU (2005 to present)? To what extent is improving country procurement capacity built in to 2.8 2.9 2.6 the country work program? To what extent have improvements in country procurement capacity 2.1 2.1 -- been tracked over time in the CAS/CASCR exercises? Source: IEG country case study questionnaire. Note: CAS = Country Assistance Strategy; CMU = Country Management Unit. 1 = Negligible; 2 = Modest; 3 = Substantial; 4 = High Box 2.1 illustrates another country (Peru) where the Bank’s country strategy documents gave significant attention to procurement. In Ethiopia, strengthening country procurement systems was reported to be an element of both the previous as well as the most recent CAS (2012). The evolution of progress achieved through the CAS has also been tracked over time. In Tanzania, strengthening of the country procurement system was reported to be a major element of Bank country strategy, specifically, an important element of the CAS pillar on governance and accountability. Most poverty reduction support credits (PRSCs) have had strong elements of procurement-related conditionality. In Azerbaijan, strengthening the country procurement system is an element of the Bank’s past country strategy, although no procurement-related indicator was included in the country partnership strategy (CPS) results framework. The upcoming CPS interim report also features procurement reform as a core area for policy dialogue. Strengthening the country’s procurement system was reported to be a strategic goal for the Bank in Mexico, of importance in support of Mexico’s e-procurement system (CompraNet), which includes improving the procurement legislation of Mexico and establishing a dedicated procurement policy unit in the government. Although it is true that Mexico launched the CompraNet e-procurement system in 1996 without the Bank´s intervention, the Bank provided advice on improving the system so that it was acceptable for use in Bank-financed procurement. 17 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Box 2.1. Peru: Procurement in the Country Partnership Strategy  Strengthening the country procurement system has been an important element of the country strategy for years, and it is specifically included in the 2007–11 CPS. Although the CPS recognized the advances made by the local authorities in terms of modernizing the country’s procurement legal framework and electronic system, it also recognized the need for increased efforts to implement the action plans of the CPAR completed in 2001, as well as the need for an updated (self) assessment to be carried out by the country in partnership with the Bank and the Inter- American Development Bank.  Advisory and non-lending technical assistance activities were part of the CPS and are also incorporated in the subsequent CPS for the 2012–16 period. Limited procurement implementation capacity has been a serious challenge both at the central and regional levels. Coupled with decentralization to regional and municipal governments, it led to the establishment of more than 1,830 procuring entities nationwide. Training is an ongoing necessity because of high staff turnover.  Diagnostic work revealed challenges, some caused by Peru’s strict and control-oriented legal framework.  A 2007 DPL did have a procurement component to support legal reform, and the implementation of an electronic portal. The drafting of an overall national procurement strategy had been supported; this appears to be finally implemented in 2013. Source: IEG field visits. By contrast, in three other countries that IEG visited—Morocco, the Philippines, and Turkey—CASs did not feature support for procurement reform or the development of procurement as important activities to be supported by the Bank. Nevertheless, in Morocco, strengthening procurement was discussed frequently with the authorities, and several IDF grants were made specifically to support procurement system reforms and to strengthen the capacity of the national procurement agency. Similarly, the Bank has made significant contributions to capacity building in the Philippines, including participation in a 2012 CPAR.28 Turkey, another middle-income country, presents a contrast to Peru and Morocco. Procurement reform ceased to be an element of the Bank’s country strategy after 2005, although it had been very much a part of the strategy following the 2001 CPAR. The 2012–15 CPS had just a few lines on public procurement regarding the issue of debarment of firms. Both country management and Bank procurement staff concurred that modest attention had been paid to procurement capacity building of late, perhaps because of the successes achieved and the perception that country systems, in many respects, were greatly improved, but also because Turkey, as a European Union accession candidate country, has a focus on alignment with the European Union public procurement directives, more than with the World Bank. 18 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Country Resources Assigned to Procurement: Lending and Capacity Building Tracing the extent to which countries have supported procurement in general, and capacity building in particular, through work program allocations is not easy because of differences, until recently, in regional practices in terms of allocation (Appendix tables B.1-B.4 have details on available budget data). On a regional basis, transfers by country departments to regional operational support units, for support to procurement in lending and supervision, are detailed in Table 2.10. To the extent that procurement supervision budgets often incorporate training and “just-in-time” advice to clients and hence serve as a vehicle of capacity building, it is reassuring that supervision resources are typically three to four times as large as resources provided during project preparation.29 It is also clear, however, that resources spent on procurement-related AAA are a fraction of the outlays that support lending and moreover have shown a distinct downward trend since 2007, especially with regard to ESW (Table 2.11). Although expenditures on non-lending technical assistance (NLTA) have been large in specific years, there is no clear trend change. Table 2.10. Project-Related Procurement Costs by Region, FY11 ($) FY11 AFR EAP ECA LCR MNA SAR Total Lending 1,242,710 478,501 589,840 382,047 1,193,617 3,886,715 Supervision 3,897,783 2,839,722 168,426 2,585,689 1,055,461 3,203,998 13,751,078 L/S (%) 32 17 23 36 37 28 Source: World Bank data. Note: LS = lending/supervision. Regions: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. Table 2.11. World Bank Procurement AAA Costs by Region and Year, FY02–11 No. of No. of ESW NLTA Total AAA FY ESW NLTA % ESW % NLTA ($ millions) ($ millions) ($ millions) products products 2002 17 0 1.7 0 1.7 100 0 2003 27 3 3.2 0.1 3.4 96.1 3.9 2004 27 4 2.7 0.6 3.3 82.2 17.8 2005 18 2 2.0 0.1 2.1 93.7 6.3 2006 12 5 3.3 0.2 3.5 93.7 6.3 2007 22 3 3.7 0.0 3.7 99.8 0.2 2008 12 8 1.9 0.4 2.3 81.6 18.4 2009 7 5 1.1 0.3 1.4 76.3 23.7 2010 7 5 0.7 1.6a 2.3 31.1 68.9 2011 6 6 0.4 0.3 0.7 59.0 41.0 Total 155 41 20.7 3.7 24.4 84.7 15.3 Source: World Bank data. 19 CHAPTER 2 SUPPORT FOR PROCUREMENT REFORM IN COUNTRY STRATEGIES Note: Total AAA is defined as ESW plus NLTA. AAA = analytical and advisory assistance; ESW = economic and sector work; NLTA = non-lending technical assistance. In 2010, a single unusual NLTA for Liberia accounted for $1.2 million. IEG also attempted, through its case studies, to obtain information on resources at the level of country work program agreements, a key instrument for implementation of a country strategy. There is little case study evidence on the division of work program agreements between building country procurement capacity and systems and support to procurement in project implementation. Among case study countries, only in Ethiopia was IEG able to obtain data on the overall resource transfer from the country work program agreements to procurement specialists separately for support to project implementation and for assistance with developing the country’s own procurement capacity. Such information is valuable for targeted and monitored programs of intervention. 20 3. Supporting Capacity through AAA—CPARs, MAPS, and PEFA Tools A prominent part of Bank support for building countries’ procurement systems was its advisory service, in the form of fiduciary diagnostics of national procurement systems. These systematic assessments were used to underpin strategic plans for the strengthening of national procurement systems. The traditional instrument the Bank used for this was the CPAR, especially in the first half of the period under review. In the latter period, 2006–11, new diagnostic instruments were introduced, notably the MAPS tool, designed jointly with OECD- DAC and partner agencies also extensively used for procurement diagnostic work by Bank clients. IEG first evaluated the quality of diagnostic reports prepared and the evolution of their use over time.30 Next, it examined the results of Bank-led diagnostic work.31 Attribution of results to the Bank is challenging, especially in a desk review, given the presence of multiple development partners focusing on procurement reform in any single country, as well as efforts made by each country itself. IEG therefore assessed outcomes in a simple manner. Where the first report in a series over time included a specific recommendation and progress was noted in the area in the second report (CPAR, MAPS, or PEFA), IEG has assumed that the Bank contributed to the progress. Apart from the limitation that this is a partial approach, a second limitation is that the investigation of any country is only as up to date as the latest available diagnostic report available. Results should therefore be read in conjunction with those from analyses of capacity building efforts through country strategies (Chapter 2) and through lending (Chapter 4). Further, perceptions of Bank advisory support, gained from field visits, are triangulated with findings emerging from desk analyses. Main Findings—Quality and Results of Bank Advisory Work  The Bank made substantial efforts to build procurement systems starting in the early 2000s, first by undertaking CPARs to review the legal and institutional framework for procurement and recommend reforms.  The first generation of CPARs included mandated instruments and was universally appreciated by client countries as comprehensive, thorough analyses, defining roadmaps for procurement reform going forward. 21 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS  The volume of diagnostic and advisory assistance for procurement declined after CPARs ceased to be mandatory. Although partially counterbalanced by the increased use of the OECD-DAC MAPS, some clients perceive the Bank’s procurement work as having lost some traction.  Although there is much overlap in core areas, CPARs included areas intended for Bank-country dialogue (notably, action plans for improving procurement) and provided flexibility to address country-specific issues. Although MAPS exercises had more consistent structure, they provided a snapshot of the system rather than a roadmap for reform. Recent hybrid exercises appear to substantially embed the best of both.  More than half the reports were found to be well structured, but with some shortcomings. CPARs were more prone to scattered discussion, sometimes of critical topics; MAPS reports lagged in terms of preparing clear and actionable strategies. And overall, reports were supported by little data.  Both CPARs and MAPS focused more on legal and institutional frameworks, sometimes in a formalistic way, and were thus less likely to take account of whether the laws were published and available to interested parties. Risk assessments were limited in both MAPS and CPARs. Legislation related to corruption tended to be covered, but there was little discussion of the existence or application of enforcement mechanisms.  Earlier CPARs had less coverage of the integration of procurement into the budgeting, planning, and audit process and the existence and dissemination of monitoring and procurement statistics, areas consistently covered by the MAPS framework. Coverage improved significantly in later CPARs (see Box 3.1).  In terms of results, in most countries at least some actions were taken based on the recommendations of Bank diagnostic work, with high or substantial implementation in around two-fifths of countries surveyed. Comprehensive diagnostics, such as CPARs and MAPs, were more important for results in procurement than PEFAs or public expenditure reviews (PERs).  The Bank tended to focus on “first-generation,” formal results, especially in the areas of legislative, regulatory, and institutional reform, not necessarily with commensurate functional value. However, political economy matters. Although the legislature may have passed laws for procurement, it may not represent wide stakeholder consensus.  In many countries where the Bank’s work was more influential, there may have been greater country capacity (Mauritius, Mexico, Morocco, the Philippines, and Turkey). Yet government commitment was also critical and sometimes governments took the lead role. 22 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS  Human capacity development focused mostly on needs for implementing Bank lending. Only a few programs looked at longer-term sustainability, for example, the lack of recognition of procurement as a professional stream. And endemic country issues affect outcomes as much Bank design: stretched resources, low civil service salaries, and high staff turnover. Procurement capacity is difficult to develop in isolation of the overall civil service cadre.  In some countries, governments have progressed toward greater integration of procurement and overall public sector management. The information on results offered in this analysis is limited by the lack of quantitative information or statistical data on procurement compliance and performance. Another limitation, particularly in decentralized countries such as Mexico and Vietnam, is that assessments are typically limited to the national level. Bank internal records also make it difficult to assess the quality of the process, the outputs, and the impact of some diagnostic work, especially PEFAs, PERs, and NLTA. Many project files lack concept notes and completion reports. To ensure accountability, specification and tracking of the usage of reporting instruments is suggested. Box 3.1. Previous IEG Findings on Procurement from IEG’s Fiduciary Diagnostics Evaluation IEG undertook a previous evaluation of the Bank’s procurement diagnostics: The Country Financial Accountability Assessments and Country Procurement Assessment Reports: How Effective Are World Bank Fiduciary Diagnostics? The report pointed out that CPARs and country financial accountability assessments, especially the latter, had a limited influence on financial management and procurement arrangements of Bank assistance, although they did influence the volume of Bank assistance. The study highlighted the fragmented treatment of overall fiduciary risk by the Bank in client countries and recommended that fiduciary instruments use an integrated risk analytical framework. The present evaluation in contrast focuses on the impact of CPARs and MAPS on building country systems. Source: IEG 2008a. CPAR, MAPS, and PEFAs—An Introduction The CPAR was initiated in the mid-1980s; it because a tool that was based on a broadly standardized approach and methodology, for Bank staff to identify national procurement practices that could not be used for Bank-financed projects. The instrument was revised in 1998, with a view to assessing the performance of countries’ public procurement systems, to initiate a dialogue and action plan with the government to strengthen these systems. From July 1999 to December 2004, the 23 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS CPAR was part of a set of core diagnostics mandatory for all active Bank clients, and a CPAR had to be completed for each at least once every five years. The most recent methodological guide to the CPAR is from 2002 (World Bank 2002b). Subsequently, they ceased to be mandated, so fewer CPARs were undertaken. Between 2002 and 2006, 89 CPARs were undertaken, averaging about 15 per year. From FY05 onward, CPARs ceased to be mandated, though another 26 CPARs (or about four per year) have been undertaken since FY06 (Box 3.2). Box 3.2. Country Procurement Assessment Reviews CPARs served a dual role. First, the fiduciary function ensures that national procurement procedures are acceptable for use in Bank-financed investment projects, helping to establish review thresholds, assess capacity of implementing agencies, guide the frequency and scope of procurement audits, and determine procurement services to be provided from field offices. An important element of the CPAR is the national competitive bidding checklist, which assesses country procedures for acceptability on Bank-financed contracts. CPAR guidelines anticipate integration with other Bank ESW products, naming specifically the PER, Country Financial Accountability Assessment, and Institutional and Governance Reviews. Second, CPARs provided the basis for capacity-building programs. The 2002 methodological guide for the first time drew attention to the role of fiduciary diagnostics as a tool for giving feedback to clients in parallel with other Bank diagnostics, such as the Country Financial Accountability Assessment. This enabled a dialogue with the government on the health of the existing system and produced an action plan to guide procurement reform. Specifically, CPARs were intended to help countries (i) increase their capacity to plan, manage, and monitor the procurement process effectively; (ii) improve the accountability, integrity, and transparency of the process and reduce the scope for corruption; and (iii) be consistent with internationally accepted principles and practices. The Bank prepared a standardized guide and set of questions intended to guide CPARs; however, the CPAR evolved to focus on country-specific circumstances and requirements, with their frequency, nature, and scope based on the type and level of Bank engagement, country priorities, and the availability of information from other development institutions and the country itself. Recent CPARs have expanded their scope beyond their originally envisaged form to frequently include the OECD-DAC MAPS assessment tool. Source: IEG assessment of revised CPAR Guidelines, 2002. CPARS VERSUS MAPS Although the CPAR declined in use, the MAPS tool (Box 3.3) was developed in parallel by the Working Group on Aid Harmonization in response to the 2005 Paris Declaration for Aid Effectiveness. The Declaration gave prominence to the need for country-level harmonization of aid arrangements of different donors, particularly with regard to procurement. The Bank was a significant participant in the process 24 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS and contributed to the design of the new product. Introduced in mid-2006, the MAPS tool was applied to an estimated 60 countries; of these, around 22 are reported on the OECD website. MAPS exercises are also increasingly embedded in recent Bank CPARs, though some are also stand-alone assessments. Yet MAPS was designed to be applied not only by the Bank or other IFIs, but also by other actors, including the country government itself for a self-assessment. It has been utilized in a variety of contexts (Table 3.1).32 There is much overlap in substantive coverage of core areas of procurement between CPARs and MAPS, though there are some areas—for example, trade practices—that are included in the CPAR but not in MAPS.33 CPAR guidelines also provide for areas intended for Bank country dialogue or Bank lending, including action plans for improving procurement, risk assessments, and so forth. CPAR guidelines provide more procedural guidance, for example, consultation mechanisms with interested donor agencies, cost estimates, and possible sequencing of reforms. However, guidance on the scoring of specific areas is far less detailed than for the checklists of MAPS. Although the less-defined structure of the CPARs led to weaker coverage in some instances, as discussed below, it also had the advantage of allowing focus on country-specific areas when appropriate. The Honduras 2005 CPAR, for example, contained a major special section on its Efficiency and Transparency in Government Program. The Rwanda 2004 CPAR discussed fiscal aspects, specifically the tax revenue foregone because of tax exemption for donor-financed contracts (World Bank 2004). The MAPS methodology also included Compliance Performance Indicators (CPIs). Unlike its core baseline indicators, which take a “snapshot” of the existing regulatory framework and institutional and operational arrangements, CPIs rely on data obtained from samples of transactions and interviews or surveys with stakeholders in the procurement system. CPIs were meant to measure the reality of a procurement system on the ground. They were presented as a complementary but not mandatory part of the MAPS exercise. In practice they were rarely used, largely because of difficulties obtaining data, some of which are simply not available in some countries. The difficulty in application of compliance/performance indicators was a noted weakness of MAPS and an area for improvement if the vitality of the tool is to be restored. However, despite countries’ reluctance to use the CPIs as laid out in MAPS, countries have increasingly recognized the need for such performance 25 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS measures, and several (for example, Ethiopia, Indonesia, and the Philippines), have devised their own methods for doing so. Box 3.3. Methodology for Assessing Country Procurement System The MAPS tool was intended to provide a common tool that developing countries and donors could use to assess the quality and effectiveness of national procurement systems. The methodology has capacity development as a fundamental objective and provides a vehicle for looking at procurement in the wider context of the legal and economic system. With MAPS, procurement systems are gauged against benchmarks in four areas or “pillars”: (i) legal and regulatory framework, (ii) institutional architecture, (iii) operations management, and (iv) independent oversight. These indicators follow from the original CPAR procedures but provide much more detailed requirements, including a scoring system for quantitative analysis. Additionally, since about 2006, the MAPS methodology has introduced semiquantitative compliance measures. With its 4 pillars, 12 indicators, 54 subindicators, point scoring system, and extensive manuals, MAPS is a more detailed analytical tool than the CPAR. It covers the same topics as the CPAR and since its introduction has been used to address the country assessment component of a CPAR. However, MAPS is not a Bank-specific tool and hence does not address Bank-specific concerns. For example, the CPAR ranks procurement risks in a country and links this to Bank operations (levels of review, for example). Similarly, the CPAR checklist compares country procedures against Bank requirements. MAPS, in contrast, was designed to be applied by different users, including as a self-assessment, but can also be used within a CPAR. As a stand-alone assessment, MAPS is of less use to the Bank than a full CPAR. The focus in MAPS on indicators and their scores is limited to assessing formal structures of a country’s procurement system and has more limited application to how well it performs in practice. The four pillars of the MAPS’ baseline indicators are:  Public Procurement Legislative and Regulatory Framework (Pillar 1)--Including implementing regulations and bidding and contract documents  Public Procurement Institutional Framework (Pillar 2)--Mainstreaming and integration of the procurement system into the public sector governance system; separation of policy from transactions to avoid conflict of interest (functional regulatory body); and institutional development capacity to undertake data collection, and analysis, training, staff development, and performance measurement  Efficient Procurement Operations and Practice (Pillar 3)--Involving capacity and competence of staff, record keeping; functionality of the public procurement market, ensuring competition and open access; and existence of contract administration and dispute resolution mechanisms  Integrity of the system (Pillar 4)--Effective control and audit systems; efficient appeals mechanisms; access to information; and existence of ethics and anticorruption measures. Sources: MAPS; OECD/DAC Working Party on Aid Effectiveness Task Force on Procurement, February 2010; Busan Partnership for Effective Development Co-operation, Strengthening Country Systems: Results and Opportunities. 4th High Level Forum on Aid Effectiveness, Busan, Korea, November 29-December 1, 2011; OECD 2006b; World Bank 2008b. Note: MAPS indicators are scored on a scale of 0–3. There are two subindicators, however, where only a score of 3 or 0 is possible: 4(d), regarding the regulatory body, and 10(e) on the existence of a complaints review body. 26 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS The recent integration of MAPS into CPARs has produced some of the highest quality CPARs. Six of the 37 CPARs that IEG reviewed integrated the MAPS baseline indicators, all from 2008 or later. The combination of these two instruments stemmed from a desire for the dependable scope of coverage from the MAPS indicators as well as the flexibility and more nuanced analysis afforded by CPARs (Box 3.4). Box 3.4. Integration of CPAR and MAPS Methodologies The perceived strengths and limitations of both instruments are illustrated by the following explanation from the 2012 Ethiopia CPAR incorporating the MAPS indicators: …To facilitate the subsequent use of the OECD-DAC assessment results in the broader capacity development process, the diagnostic approach of the OECD-DAC framework was widened to also include an explanatory dimension. In this way, the focus of the data collection was not only on “which” systems are in place and “how” these work in practice, but also on “why” this is the case, hereby providing an important starting point when defining actual capacity development strategies later on in the capacity development process…. The Moldova 2010 CPAR was another report that drew on both frameworks. In addition to a comprehensive snapshot of the basic facets of the procurement system guided by the baseline indicators, it contained a major additional section on procurement practices in the health, transport, and education sectors. Of all the reports in IEG’s sample, the recent CPARs integrating MAPS were consistently the highest in quality and most comprehensive in scope. Sources: Republic of Moldova CPAR (2010); Ethiopia CPAR (2012). Finally, there was also a move to undertake more integrated diagnostics of public procurement and public financial management systems, given their close interlinkage within the wider framework of public financial management and budgetary processes. The Bank’s Public Expenditure and Financial Accountability Secretariat developed a diagnostic tool that included a specific indicator on public procurement. 27 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Table 3.1. Procurement Diagnostic Products, FY02–12 Product 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 Total Bank MAPS 6 2 6 14 Other MAPSa 2 11 8 1 22 PEFAb 19 15 21 20 16 10 1 102 CPAR 14 25 25 16 5 4 6 2 3 2 2 104 Total 14 25 25 35 22 36 34 24 16 9 2 242 Sources: IEG analysis, PEFA secretariat. Note: CPAR = Country Procurement Assessment Report; MAPS = methodology for assessing procurement systems; PEFA = Public Expenditure and Financial Accountability Assessment. a. Includes only those available on the OECD website. However, up to 60 MAPs exercises have been cited as available, sometimes undertaken by countries themselves, sometimes embedded within exercises such as recent CPARs. The years shown refer to the calendar year in which the report was published. b. Includes only reports whose preparation was either led by the World Bank or involved Bank participation. The years of the PEFA reports refer to the timing of the main mission during which most interviews were conducted. Only finalized reports (both published and unpublished) are included. Does not include PEFA reports of subnational governments. PUBLIC SECTOR MANAGEMENT AND THE PEFA INSTRUMENT Recognizing the importance of public sector procurement as a key element in overall public sector administration, the Bank has made efforts to integrate elements of its advisory work on procurement within a wider framework of public expenditure management. One key aspect of its work in this area was the building of an integrated diagnostic tool, the PEFA assessment tool (Box 3.5). Although CPARs and MAPs are focused more specifically on procurement systems, the Bank and other development partners developed the PEFA as a broader diagnostic instrument for overall public financial management. Other elements of Bank analytic work, as well as lending, can serve to integrate procurement within the wider scope of public sector management. For example, the Bank also undertakes PERs that could include, in principle, reviews of the efficiency of public expenditures and the modes in which they are planned and undertaken.34 28 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Box 3.5. Building Capacity Through Use of the PEFA Instrument The PEFA framework was developed by a group of seven international organizations to assess whether a government has systems to deliver aggregate fiscal discipline, strategic resource allocation, and efficient use of resources for service delivery.a PEFAs are normally undertaken by the Bank or one of its partners, preceded and followed by workshops with the client government. In a limited number of cases, governments have carried out their own self-assessments using the PEFA methodology. Drafts are peer reviewed by the PEFA Secretariat. (http://pefa.org/en/content/pefa-check). The purpose of the PEFA framework is to  Measure the performance of public financial management and procurement systems, processes and institutions  Give a reliable indication of whether government reforms are improving performance, and if so, what can be learned for other countries addressing the same challenges  Contribute to a harmonized approach to donor support around a common assessment framework to reduce transaction costs for governments. PEFA has one indicator out of 28 measures of government capacity that directly relates to procurement. Indicator PI-19 initially covered three simple dimensions and was subsequently revised to cover four dimensions: the legal and regulatory framework for procurement, public access to procurement information, the extent of open competition, and the use of a complaints mechanism. The current version has several subindicators under each dimension. Other PEFA indicators also inform aspects of procurement, via the control environment, including sound budget formulation, transparent public information, internal controls in spending ministries, timely in-year budget reports, and internal and external audits. Indicators to measure these aspects include measuring the monitoring of expenditure payment arrears, and seven other indicators that measure other public financial management processes with links to procurement. It is recognized that PI-19 is a very limited, high-level measure of the extent of improvement of inputs in public procurement. Moreover, PEFA does not measure procurement performance, for example, the scale of procurement (procurement budget relative to total budget), many key elements of the quality of contracting (average contract value, extent of use of e-procurement, proportion of procurement accredited staff in tender committees, bunching of contracts and unspent funds at end of year, market domination by small number of contractors), quality of outcome (completion rates, unit costs, changes in service quality), and equity of the process (citizen participation, extent of irregularities, and application of enforcement mechanisms). However, gathering information on some of these aspects may be a challenge today in many Bank borrowing countries. Sources: PEFA Secretariat; The Asia Foundation; IEG interviews. a. The World Bank, the International Monetary Fund, the European Commission, the United Kingdom’s Department for International Development, the French Ministry of Foreign Affairs, the Royal Norwegian Ministry of Foreign Affairs, and the Swiss State Secretariat for Economic Affairs. 29 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS IEG Evaluation of Procurement Diagnostics—Methodology and Approach IEG’s evaluation of these reports is based on a standardized template structured around the core pillars of the CPAR and MAPS (Appendix Table C.1).35 Where a MAPS assessment was conducted by some other actor, whether a self-assessment or by another donor, IEG analyzed findings but did not include its scores in any aggregates. In total, the sample included 49 reports from 32 countries, including 37 CPARs (including 5 with integrated MAPS), 3 Country Procurement Issues Papers, and 9 self-standing MAPS exercises (Table 3.2). IEG also referenced six MAPS assessments from non-Bank authors (Box 3.6). The first part of the review examines the quality of inputs and processes; the second part reviews results achieved over the time frame of the evaluation. Table 3.2. ESW Reviewed by Product Type, Region, and FY FY of By product type By region Total for completion CPAR CPIP MAPS AFR EAP ECA LAC MENA SAR each FY 2000 2 1 1 2 2001 4 1 2 1 4 2002 8 2 1 2 3 8 2003 7 4 3 7 2004 1 2 3 3 2005 4 1 1 1 1 4 2006 2 2 2 2007 1 1 1 2008 (2) 1 1 2 3 2009 1 4 3 1 1 5 2010 1 3 2 1 1 4 2011 (1) 2 1 1 1 3 2012 (3) 2 1 3 Total 37 3 9 16 8 6 9 5 5 49 Source: IEG analysis. Note: The CPAR reports shown in parentheses are those for which the MAPS methodology was applied. CPAR = Country Procurement Assessment Report; CPIP = Country Procurement Issues Paper; MAPS = methodology for assessing procurement systems. Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; SAR = South Asia. STAKEHOLDER PARTICIPATION IN ESW PREPARATION Greater involvement and ownership by stakeholders in the procurement process, whether host governments, private sector suppliers, or observers in civil society, has been repeatedly identified as a factor leading to more effective diagnostic work (OECD 2009; IEG 2008a). The first part of IEG’s analysis reviews available evidence on the extent to which client governments, suppliers, other donors (including IFIs and bilateral donors), and other stakeholders were involved in the diagnostic process (Table 3.3). Evidence suggests a high level of participation by government 30 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS counterparts and frequent participation of other partners in development. Civil society was less involved, but this could reflect the relative scarcity of such groups in many client countries.36 Table 3.3. Case Study Evidence: Country and Partner Ownership and Participation in ESW Average score all Questions on stakeholder participation countries CPAR MAPS To what extent did the client stakeholder/government participate in preparation of procurement ESW/AAA (especially bidders)? Is this evident 2.5 2.6 2.1 from the report? To what extent were civil society, academia, NGOs, and media involved? 0.9 0.9 0.8 To what extent did other development partners participate? 1.4 1.7 0.4 Source: IEG desk analysis. Note: Score based on four-point scale, 0–3. AAA = analytic and advisory assistance; CPAR = Country Procurement Assessment Report; ESW = economic and sector work; MAPS = methodology for assessing procurement systems; NGO = nongovernmental organization. Client Government Participation The reports surveyed by IEG were strong in this area overall, scoring an average of 2.7 on a four-point scale of 0–3, although there were some exceptions. Fifteen reports were conducted jointly, with a team composed of representatives from both the World Bank and the government. An additional 14 reports noted close collaboration with the client government. Even where governments were relatively less involved, counterparts often provided input through meetings (nine reports) or interviews (six reports).37 Draft reports and their action plans were frequently reviewed with the government before being finalized. Box 3.6. Self-Assessments In addition to exercises led by the World Bank, governments often utilized the OECD-DAC tool to carry out a self-assessment, either independently or in parallel to the Bank assessment. Even where the ultimate product was not high quality, the exercise proved to be a valuable tool for encouraging country ownership and engaging in dialogue. Peru, for example, specifically requested that the Bank not validate its 2008 MAPS exercise because it wanted to take the lead in its own procurement reform. The 2010 Moldova and 2011 Sierra Leone CPARs, both of which integrated the MAPS indicators, provide alternate models of parallel self-assessment. In Moldova, the CPAR authors used the country’s self-assessment conducted using the OECD-DAC methodology in 2008 as a framework, explaining reasons for any deviation. In Sierra Leone the assessments were conducted in parallel. Bank and government teams first agreed on the scope, timing, and methodology of the exercise. The Bank, the government and other stakeholders then collaborated to discuss and prepare a mutually agreed final report. Source: IEG analysis. 31 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Donor Collaboration Other development partners played a prominent role, especially in many CPARs (Table 3.4).38 The Asian Development Bank (ADB) and Inter-American Development Bank (IDB) in particular had strong roles in their respective regions, each jointly authoring or offering major contributions to five CPARs. The African Development Bank also jointly authored two CPARs with the World Bank and was present for the Senegal MAPS exercise as an observer. The Philippines 2012 CPAR, published jointly by the ADB, the government of the Philippines, the Japan International Cooperation Agency (JICA), and the World Bank, provides a case study of a high degree of collaboration producing a very high-quality report. Table 3.4. Donor Collaboration by Region Question on donor collaboration AFR EAP ECA LAC MENA SAR To what extent did other 1.2 1.9 1.7 1.8 1.4 0.8 development partners participate? (score 0–3) Source: IEG analysis. Note: Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; SAR = South Asia. Civil Society, Academia, and Media Involvement Involvement was significantly less for civil society, academia, nongovernmental organizations, and the media. Only one report, a 2001 CPAR from Indonesia, indicated participation by all of the above. Collaboration with academic institutions was noted in three cases. Civil society and nongovernmental organizations were most commonly involved in this group, with their input noted in 10 reports. Civil society representatives were often on the roster of meetings undertaken by the assessment team or participants in workshops. Notably, in the Sierra Leone 2012 CPAR, representatives from civil society carried out part of the government’s self- assessment. Quality of the Bank’s Procurement Diagnostics (2002–12) CLARITY OF OBJECTIVES, ANALYSIS, AND STRUCTURE More than half of the reports surveyed were substantially well structured and clear in their objectives and analysis (Table 3.5). Among those found lacking, CPARs and MAPS suffered from different deficiencies. As the great majority of CPARs lacked the standard organization of MAPS, they were less clear in their objectives and had scattered discussion, sometimes of critical topics. For example, the Honduras 2005 32 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS CPAR was structured with an overview of public sector procurement in Chapter 1 and a more in-depth look at the national procurement system in Chapter 2. This bifurcation meant that key areas such as the legislative framework were discussed in a two places in the report. Across all reports, somewhat less central areas of analysis, such as audits or private sector constraints, were more prone to disjointed treatment (Box 3.7). The great majority of MAPS assessments, by contrast, benefited from the rigid framework of its four pillars and underlying subindicators. This largely resulted in clear and logical organization of key areas of coverage. Although there were sometimes thin areas (Box 3.7), these failings were the exception rather than the rule, and most MAPS assessments undertaken as part of the use of country systems piloting program were found satisfactory. The MAPS for the Philippines, for example, contained recommendations as generally applicable and actionable as any CPAR, such as publishing standard processing times for each step of the payment process and adding a training program for the private sector to the national agenda. Table 3.5. Overall Report Quality Average score Questions on report quality all countries CPAR MAPS What is the overall clarity of objectives and analysis, structure of 2.7 2.7 2.6 report? Was empirical data used to support findings? 1.3 1.4 0.8 Does the ESW conduct a risk assessment of public and of Bank 1.5 1.6 1.0 procurement? What is the quality of the Action Plan included in this ESW 2.2 2.3 1.8 report? (CPAR) (score 3–0) Source: IEG analysis. Note: CPAR = Country Procurement Assessment Report; ESW = economic and sector work; MAPS = Methodology for Assessing Procurement Systems. 33 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Box 3.7. CPARs and MAPS—Strengthening Clarity of Objectives and Structure The stated objectives of early CPARs reflected a limited vision for use of the tool. In many cases the only stated objective was to review the countries’ procurement systems, with no broader context or intended outcome other than producing the report itself. The Mexico 2002 CPAR, for example, introduced the exercise saying only, “During the review of Mexico’s Country Assistance Strategy, it was agreed that a mission for the Country Procurement Assessment Report would be organized to review procurement issues.” With time, the vision in many CPARs became clearer. In contrast to the above, the 2007 Mexico CPAR aspired to ….(a) ascertain how the federal procurement and financial management systems could make a greater contribution to the efficient use of fiscal resources; (b) determine the extent to which Mexico’s public procurement system supports or hinders implementation of the government’s policy and objectives; (c) identify opportunities to reduce operating costs; and (d) identify issues that are adversely affecting economy, transparency, and optimal outcomes. It also helps the banks ascertain the extent to which they can rely on Mexico’s procurement and supervisory systems to discharge their fiduciary obligations…. Given the highly defined structure of MAPS reports, some emphasized structure over substance and readability. Reports for Mauritius and Senegal discussed most MAPS sub- indicators only superficially in a table, as opposed to more comprehensive treatment in text. The Albania 2010 MAPS produced comprehensive tables, but it was difficult to draw out an overall vision. Shortcomings in some MAPS reports may have reflected the context in which they were undertaken. Those MAPS assessments conducted as the first stage of the Bank’s use of country systems pilot program were not intended to be standalone documents. Requirements for the pilot program were at times listed as “partially” fulfilled, but for undetermined reasons and with no meaningful analysis. Some recommendations were temporary measures limited to a future phase of the pilot program. The analysis of some areas requiring improvement was deferred to the second stage of the pilot, the Consistency and Equivalence Assessment. Source: IEG analysis. Quality of Action Plan The CPAR guidelines declare that the “main purpose” of the CPAR is to “establish the need for and guide the development of an action plan” to improve the country’s procurement system, and that the CPAR itself should “develop a prioritized action plan to bring about institutional improvements” (World Bank 2002b). Although the MAPS framework concedes that an action plan may be a part of an assessment, the idea was presented as optional and an afterthought, with no guidance as to the content and form. MAPS guidelines state only, “When the report goes beyond the 34 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS mere assessment of the system into proposing an action plan or a reform strategy, the relevant sections and chapters need to be added to the report.” IEG looked to three factors in evaluating the quality of reports’ action plans: i. Were the recommendations prioritized? ii. Was there some time frame given? iii. Was there some discussion of who would be responsible for implementation? IEG also evaluated whether the recommendations were clear, actionable, and related to weaknesses identified, as opposed to a “wish list” unlikely to generate meaningful change. Examples of unsatisfactory recommendations were to “establish… a competent oversight body” (Ethiopia 2002 CPAR) or to “…enact a Public Procurement law and implementation regulations” (Indonesia 2001 CPAR). As a whole, the sample included a mix of all levels of quality of action plans. Many were found to be satisfactory in terms of containing priority, time frame, and responsible entities for implementation. Only four reports of those surveyed mentioned any consideration of the cost of implementing the action plan. The quality of action plans improved slightly in the later CPARs (Table 3.6). Table 3.6. Action Plan Quality by Time Period and Report Type Time period CPAR MAPS Total 2000–07 2.2 NA 2.2 2008–12 2.5 1.8 2.2 Total 2.3 1.8 2.2 Source: IEG desk review of procurement diagnostics. Note: Score based on four-point scale, 0–3. CPAR = Country Procurement Assessment Report; MAPS = Methodology for Assessing Procurement Systems. As might be expected from the lack of emphasis given to action plans in the MAPS framework, these reports lagged behind CPARs in this respect, averaging a score of 1.8 out of 3.0 as opposed to 2.3 for CPARs. From the point of view of preparing a clear strategy for building procurement systems and procurement capacity, this was perhaps the most significant shortcoming of the MAPS instrument. However, a part of this may again be attributed to the unique context of the MAPS assessments conducted as part of the use of country systems pilot program. Although some of the MAPS reports through the program did contain substantive and constructive recommendations for improvement, action plans focused narrowly on achieving scores required for the pilot program. 35 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Use of Empirical Data Despite the emphasis on empirical data in both the CPAR and MAPS guidelines, the reports focused more on the existence of structures and regulations and were supported by little data on how these systems worked in practice (also a finding of IEG 2008a). The CPAR guidelines underscore that a “strong analytical framework, supported by sound empirical data which is carefully tailored to the specific country context, is a prerequisite for producing a CPAR of good quality,” going on to emphasize, among other things, the use of quantitative information (World Bank 2002b). The MAPS tool, for its part, envisioned that CPIs would be integrally used along with the baseline indicators to give a complete assessment of a procurement system. The use of country systems pilot program, however, planned to include this only in stage III, which was never realized for any of the participating countries. Both the CPARs and MAPS reviewed incorporated very little empirical data. It is noteworthy that the CPARs fared slightly better, scoring an average of 1.8 out of 3, compared to 0.8 for MAPS. Part of the issue is likely to be the constraints on the availability of data. The 2001 Turkey CPAR expressly noted the limited availability of data. In interviews, World Bank staff cite this as an important reason that they did not usually undertake analyses of actual performance. There was little difference in the use of such data between earlier and later ESW. Of the nine reports to feature strong use of empirical data, five came from two countries—Mexico and the Philippines. Uganda provides another example, although its later MAPS report was a self- assessment. After a 2004 CPAR strongly supported by empirical data, the government undertook a 2007 MAPS self-assessment using the Compliance Performance Indicators. These examples suggest that once there is a precedent to collect and analyze empirical data on a procurement system, it becomes easier to do so in subsequent reports. Risk Assessment Risk assessments were another area found lacking in both MAPS and CPARs, despite the fact that the CPAR guidelines underscore the need to identify clear areas of risk and propose risk management strategies, with targeted interventions, to reduce these risks (Box 3.8). 39 The MAPS framework, in contrast, makes limited references to specific, standalone risk assessments.40 Accordingly, MAPS reports were weaker in this area, with an average score of 1.0 out of 3.0, as compared to 1.6 for CPARs, and the treatment of risk was mostly piecemeal. Most CPARs fared only marginally better. Risk assessments lacked foundation and detail, if they were undertaken at all. Where there was a risk assessment, it rarely distinguished 36 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS between the risk for Bank projects and that for public procurement generally. Only three reports of those surveyed—the Colombia 2001 CPAR, Mexico 2007 CPAR, and Vietnam 2002 CPAR—undertook a separate risk rating for Bank projects, compared to country procurement. However, four reports did use the MAPS framework to produce a risk assessment, with varying levels of effectiveness. None of these exercises was purely MAPS; they were three CPARs integrating MAPS and one Country Procurement Issue Paper. This again speaks to the absence of emphasis on risk analysis in the basic MAPS framework. Box 3.8. Risk Assessments in Diagnostic Work—MAPS, CPARs, and Integrated Reports The 2007 Indonesia MAPS self-assessment did undertake a specific risk assessment, linking each risk to a baseline indicator and proposing short- and medium-term mitigation measures. The accuracy with which this reflected the reality, however, is questionable, as risk is rated to be “average” despite serious concerns for corruption discussed in the rest of the report. In the absence of any framework for risk analysis in the MAPS guidelines, the Ghana MAPS tried to incorporate one in its discussion of audit, yet stopped short of making its own assessment. It noted that “….internal audits still focus on pre-payment audit instead of risk- based audit concentrating on systemic issues with the objectives of ensuring (1) conformity to the government’s strategy; (2) effectiveness and efficiency of operations; (3) reliability of financial reporting; and (4) compliance with applicable laws and regulations.” In terms of CPARs, the risk assessment of the Azerbaijan 2002 CPAR was typical of many: … Based on the analysis of its legislative framework, the effectiveness of its regulatory institutions, the strength of its enforcement regime, the capacity of its institutional and human resources, performance on Bank-assisted projects and the threat of corruption, the assessment found that the environment for conducting public procurement in Azerbaijan is medium- to high-risk. As regards attempts to integrate a risk assessment into the MAPS framework, the analysis in the Philippines was particularly robust. In both 2008 and 2012, the scores for each pillar and indicator were compared against a “baseline” perfect score. The Moldova 2010 CPAR adopted a similar approach, assigning risk ratings for each indicator, although without a numerical basis. The 2004 Sierra Leone Country Procurement Issue Paper referenced the MAPS framework at a broader level, declaring that “the risk rating on all four basic procurement pillars is ‘high.’” Source: IEG analysis. Legal and Regulatory Framework IEG examined four questions related to the legal and regulatory framework. In terms of coverage, this area was consistently included.41 IEG examined (i) 37 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS uniformity of application, (ii) comprehensiveness of coverage, (iii) accessibility, and (iv) clarity of the hierarchy of procurement sources. For the most part, reports at a minimum covered uniformity of application and clarity of the hierarchy, giving clear description of the laws governing public procurement and their implementing regulations.42 Legal frameworks were often judged in comparison with international/IFI best practices. However, the discussion tended to be overly formal. The important area of accessibility—that is, whether the laws were published and available to interested parties—was by far the most likely to be neglected.43 Notwithstanding this, the 2008 Peru CPAR provides a rare example of excellent coverage of accessibility as well as other topics related to the legislative framework. It notes that all laws/standards should be published in the national newspaper El Peruano, as well as its digital version, with the caveat that there is limited access in some regions. IEG next investigated the extent to which the regulatory framework clearly described available procurement methods, their applicability, and admitted exceptions. The great majority of ESW contained basic coverage of this area, concentrating especially on whether open competition was the default procurement method (to be in accordance with international best practice). Second, reports usually examined under what circumstances deviations from competition were allowed, especially to sole source procurement. Most reports went beyond formal procedures, analyzing the clarity of the applicable regulations and how they were enforced in practice. Many focused on whether value thresholds were used to determine procurement method and if so, what those thresholds were, often providing a chart. Recommendations centered around reinforcing open competition as the default procurement method. These discussions illustrate a perhaps narrow optic of procurement methods, reflecting prevailing practice in the Bank. There was almost no coverage of more innovative procurement methods, such as framework agreements, negotiation, and reverse auctions. IEG also reviewed the extent to which reports covered all key steps of the procurement process ( including procurement planning, prequalification, advertising, scope of communications during bidding, receipt of bids and bid opening, bid evaluation, contract award, and effectiveness) and the extent to which regulations were implemented, updated, and supported by standard bidding documents, guidelines, and application manuals. Coverage in both areas was again relatively strong, both averaging a score of 2.6 across all countries. Standard bidding documents were a frequent topic of interest. Many countries lacked them at the time of earlier CPARs.44 Where standard bidding documents were in place, CPARs often compared them to Bank standards and general good practice. In the case of MAPS, those countries that advanced to stage II in the use of country systems 38 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS pilot program conducted an in-depth review of bidding documents as compared with Bank guidelines.45 Institutional Framework The diagnoses gave robust coverage to normative and regulatory bodies responsible for procurement. (This received the highest average score of all indicators examined by IEG: 2.8 out of 3). Nearly all reports discussed the existence, or lack thereof, of entities responsible for various procurement functions. Similar to the legal and regulatory framework, thinner coverage took the form of providing a roster of institutions and functions and stopped short of discussing their independence and general functionality. The Lao PDR 2002 CPAR and the Morocco 2000 CPAR, both of which recite the entities in charge of procurement but provide no analysis of their effectiveness, provide a contrast to the comprehensive treatment in the Peru 2005 CPAR. The Peru CPAR not only recounted the functions of institutional actors, but discussed past achievements; public perception; and administrative, economic, and financial autonomy. Other areas examined as part of the institutional framework were the integration of procurement into the budgeting, planning, and audit process and the existence and dissemination of monitoring and procurement statistics (Table 3.7). These areas were both consistently covered by the MAPS framework (indicators 3 and 11). Somewhat neglected in earlier CPARs, coverage improved dramatically. Table 3.7. Integration of Procurement into the Budget Process and Procurement Monitoring Integration 2000–07 2008–12 Total Does the report discuss the integration of procurement into the budgeting, planning and audit process? CPAR 1.7 2.8 1.9 MAPS 3.0 3.0 Total 1.7 2.9 2.1 Does the report discuss the existence and dissemination of monitoring and procurement statistics? CPAR 1.8 2.6 1.9 MAPS 2.9 2.9 Total 1.8 2.8 2.1 Source: IEG analysis. Note: CPAR = Country Procurement Assessment Report; MAPS = Methodology for Assessing Procurement Systems. Regarding the integration of procurement with overall public sector management, there was good coverage of the extent to which procurement was delayed by issues such as budget uncertainty and disbursement delays. Regarding monitoring, MAPS reports and later CPARs generally did well in covering both the legal requirement and practical availability of monitoring and procurement statistics, in terms of recordkeeping practices, reliability, accessibility, and scope of data. 39 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS It is notable that the Bank strongly endorsed the use of electronic systems to collect and disseminate statistics, an area where the Bank itself has faced challenges.46 Rwanda, for example, was praised in its 2009 MAPS for websites offering public information related to procurement. In contrast, the 2010 Mauritius MAPS noted that although minimum standards were met, there was a need to upgrade to a fully integrated information technology–based system. Much progress has been made since earlier CPARs, such as the Morocco 2000 CPAR, which mentioned monitoring and the entities responsible for it throughout the report but did not discuss the quality and accessibility of the actual data. Nonetheless, recommendations on best practices seemed to follow a prescribed formula (electronic procurement systems, regular and comprehensive record keeping, the creation of oversight agencies, and so forth) but did not explore the practical utility of these systems for client countries. Few advanced countries today have fully integrated data systems of public procurement, much less ones that integrate procurement data with other data. Data expectations could take note of the state of practice achieved elsewhere. Capacity Levels and Capacity Development Assessment of overall staff capacity levels related to procurement and efforts in capacity building were generally undertaken, with the exception of some earlier CPARs.47 For example, the Cambodia 2004, Lao PDR 2002, and Nepal 2001 CPARs noted low capacity but did not analyze the constraints or contain constructive discussion of reform. The majority of reports found a lack of capacity on the part of procurement staff, worsened by high turnover, which was caused by the lack of established career paths in procurement. Many reports recommended a national training strategy and more priority for capacity building. However, reports fell short of specifics, especially in areas such as the financial sustainability of training or assessment of different ministries and agencies. This is despite the fact that CPAR guidelines call for “…detailed institutional analysis of key government ministries and agencies… their strengths and weaknesses, priority capacity building needs… and the incentives needed to improve performance…. ” (World Bank 2002b). Fraud and Corruption Fraud and corruption was cited by a number of reports as a subject of widespread public concern.48 IEG assessed coverage of the (i) scope of provisions related to corruption and enforcement mechanisms and effectiveness, (ii) public perceptions of public procurement integrity, and (iii) the existence of codes of ethics for the public and private sectors. Coverage of fraud and corruption was found to be varied, ranging from bare bones treatment of whether or not legislation exists (Lao PDR 2002, Mauritius 2003, and Morocco 2000 CPARs) to nuanced and comprehensive 40 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS (such as Colombia 2001 CPAR, Mozambique 2002 CPAR, and Philippines 2008 CPAR). The majority of reports did a comprehensive survey of legislation related to corruption, identifying provisions (or the lack thereof) on fraud, conflicts of interest, and collusion. Others, though they discussed some aspects of corruption, neglected to mention the existence or application of enforcement mechanisms such as disqualification, suspension, and debarment, which are critical to understanding how corruption is managed within the procurement system.49 The Colombia 2001 CPAR used corruption as a framework for a larger section that touched on many related areas. It first surveyed the reality of corruption in procurement, citing a report that interviewed private contractors, then discussed relevant institutions and rules. The report also identified and analyzed failings in other aspects of the system that it concluded were related to high levels of corruption, mainly ineffective internal audits but also a lack of clarity in some aspects of the procurement law. Finally, it briefly recounted government actions to combat corruption. This report serves as an example of how the more flexible structure of the CPAR can sometimes be advantageous for a cross-cutting issue that cannot be effectively addressed in isolation. Public Procurement Transactions, Complaints, Contract Administration, and Audits Issues related to the performance of the public procurement market were relatively neglected by earlier CPARs but in later reports were for the most part given due importance by the MAPS framework (Table 3.8). CPAR guidelines give no effective framework to assess the functionality and efficiency of the public procurement market, whereas MAPS indicator 7 (functionality of the public procurement market) mandates attention to relevant factors, such as mechanisms for PPPs and other aspects of market access by private sector institutions. Only a few reports discussed the important topic of the quality of procurement outcomes. One good example is the Honduras 2005 CPAR, which stated that ...The government, donors and the private sector have gained more confidence by the relative transparency, efficiency and economy offered with these arrangements. For instance, bidding turnover has been reduced from 14 to 5 months in some areas; competition has increased, reflected in the number of participants in the bidding process and the number of firms awarded contracts. Changes such as these have generated substantial savings.... 41 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Table 3.8. Transaction Efficiency, Complaints, Contract Administration, and Audits Time period 2000–07 2008–12 Total To what extent does the report assess the efficiency of public procurement transactions and quality of outcomes? CPAR 1.7 3.0 2.0 MAPS 2.7 2.7 Total 1.7 2.8 2.1 To what extent does the ESW assess the complaints process and the appeal mechanism? CPAR 2.1 3.0 2.3 MAPS 2.8 2.8 Total 2.1 2.9 2.4 To what extent does the ESW cover issues related to contract administration and management? CPAR 1.6 2.9 1.9 MAPS 2.3 2.3 Total 1.6 2.6 2.0 To what extent does the report examine how internal and external audits and controls are conducted? CPAR 2.1 3.0 2.3 MAPS 2.9 2.9 Total 2.1 2.9 2.4 Source: IEG analysis. Note: CPAR = Country Procurement Assessment Report; ESW = economic and sector work; MAPS = methodology for assessing procurement systems Coverage of the complaints process and the appeal mechanism varied greatly in quality as well as in the subjects covered, although the lack of an appeal process or proper complaint mechanism was a recurrent theme. The Ethiopia 2012 CPAR noted that the lack of a procurement complaints board was “a matter of a serious concern as the lack of a proper complaints function is a major obstacle to fair and transparent procurement processes ….” The Honduras 2005 CPAR acknowledged that the law did provide limited procedures to request clarifications on bidding documents and address complaints. It also noted, “Traditionally, complaints do not succeed (even when they may have solid grounds)...because the executing units usually do not follow up on the claims and they delay the final response to the process, which is normally not affected by the complaint.” Discussion of contract administration and management clustered around challenges stemming from low staff capacity (attributed in at least two reports to a lack of career track for procurement) and administrative delays. In an unusual use of empirical data, the Sierra Leone 2004 CPAR noted that “of the 75 contracts assessed, 41 contracts (55 percent) were delayed in completion, most of them under goods contracts.” Causes included “delays in the release of funds, delays in establishing Letters of Credit, general inefficiency of contractors and suppliers and poor supervision by government staff.” However, there was no mention in this context of fair handling and resolution of disputes. All reports surveyed had some discussion of audits and controls, a “powerful instrument to minimize leakage of funds on 42 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS contracts” although the CPAR framework in this area is minimal.50 MAPS provided more structure through a dedicated set of indicators on audit and control systems. E-Procurement, Subnational Issues, and Green Procurement Finally, IEG also briefly surveyed coverage of some special topics: electronic procurement (“e-procurement”), subnational or local government issues, sustainability, and “green” procurement. Electronic or “e-procurement” received the most attention among these topics. There was clear consensus regarding its increasing importance. At least seven reports included recommendations on introducing and/or strengthening e- procurement as a powerful tool for improving country systems. Electronic information systems and publication, as well as legal provisions for e-procurement, were assessed and encouraged as a gateway to fully electronic procurement. In cases where e-procurement had been successfully instituted, such as the Philippines, it was praised for bringing the system “into the digital age and vastly improving transparency and efficiency.” It is not uncommon for countries to have different procurement schemes in place at the subnational/local government level, and this was a theme that emerged in 26 of the 49 reports. In some cases, the subnational procurement system was so developed that it merited its own version of the CPAR. MAPS assessments were conducted as part of the use of country systems pilot program for two subnational entities (although not included in the scope of this review): the Brazilian state of São Paulo and the state-owned corporation Power Grid of India. In Mexico, the Bank conducted individual CPARs of 4 of the 32 states. In Ethiopia, the 2012 CPAR was expanded to include “mini CPARs” of the 9 regional states and 2 city administrations. A number of reports noted lower capacity and higher fiduciary risk at the state level. Increased risk at the local level and recommendations to address it were noted in the Albania 2001, Mozambique 2002, Indonesia 2001, Peru 2005, and Rwanda 2004 CPARs. However, potential benefits in the form of transparency and accountability with greater community participation were also acknowledged. There was almost no coverage of sustainability and “green” procurement, despite the fact that this was highlighted as an emerging area by the CPAR guidelines. Procurement-Related NLTAs It should be mentioned that apart from the major elements of analytic and advisory work, through such tools as the CPAR and MAPS, the Bank also provided minor 43 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS funding through NLTA for procurement reform; this was typically focused on the preparation of special studies, legal documents, or investigative reviews of select procurement related issues. IEG analyzed six procurement-related NLTAs, drawn from five countries, identified as potentially focusing on procurement issues.51 Overall, targeted outputs were typically achieved, though there is limited information on their eventual outcomes or long-run impact. IEG’s review of selected NLTAs is detailed in Appendix C. The assessment focuses on the quality of the process and outputs of each NLTA and presents limited available evidence on possible impact. The review suggests that these highly focused short inputs were often useful in project preparation, though in some cases their offtake is difficult to trace, and long-run impact is little known or documented. Quality of Results Achieved Through the Bank’s Procurement Diagnostics IEG undertook a desk review to assess the extent to which CPARs and MAPS and their action plans contributed to results in procurement reform (Table 3.9). IEG conducted this assessment for 18 countries, where there were at least two analytical pieces (a CPAR or a MAPS) during the evaluation period.52To the extent available, PEFAs were also referenced to validate results.53As with the quality assessment, IEG used the MAPS structure of four pillars: legislative and regulatory framework; institutional framework and management capacity; procurement operations and market practice; and integrity of the public procurement system, based on a four- point rating scale.54 If a recommendation was noted as an action undertaken in a subsequent report, IEG scored this as an indication of progress. A limitation of this form of analysis is that results are only as up to date as the last available diagnostic report on any country. Results should therefore be reviewed in conjunction with results from the rest of the analysis, and from field visits. IEG’s desk review suggests mixed results in terms of the influence of the Bank’s AAA work in encouraging procurement reform (Figure 3.2 and Figure 3.3). Success was focused on “first-generation” results in the earlier part of the period and based particularly on CPARs. Based on IEG ratings, action plans were implemented to a high or substantial extent in 7 (two-fifths) of the 18 countries. The influence was modest in another 10 and low in 1 (Figure 3.1). In many of the countries where the results were modest, such as Ethiopia, Tanzania, and Uganda, the focus of the action plans was on so-called “second-generation reforms” during most of the evaluation period. Therefore, admittedly, the reform task has been more challenging. 44 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Table 3.9. IEG Analysis of Results—Fiscal Years of CPARs/MAPS Reviewed Fiscal year  Country  2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2012 Albania             Azerbaijan   Colombia             Ethiopia   Ghana             Indonesia   Lao PDR             Mauritius   Mexico             Morocco   Philippines, The            Rwanda   Sierra Leone             Tanzania   Turkey             Uganda   Vietnam   West Bank                      and Gaza Total 1 5 6 3 3 2 1 3 4 3 4 3 Source: IEG review. Figure 3.1. Overall Levels of Influence of Bank ESW Source: IEG. Note: ESW = economic and sector work. 45 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Figure 3.2. Overall Levels of Improvements in Each Dimension Source: IEG. Bank advisory work has had high or substantial impact on the first two dimensions of procurement reform—legal and institutional frameworks—for at least half of the public procurement systems reviewed. However, better market development and effective internal and external control frameworks have been more challenging to achieve, especially the last, particularly in contexts of weak capacity and governance frameworks (Figure 3.2). Figure 3.3. Improved Procurement Reform Supported by Bank AAA Rwanda 2.57 Turkey 2.48 Morocco 1.80 Phillipines 1.72 Mauritius 1.71 Tanzania 1.59 Ghana 1.50 Albania 1.33 Vietnam 1.32 Colombia 1.24 Uganda 1.16 Sierra Leone 1.13 Mexico 1.04 Ethiopia 0.97 Azerbaijan 0.93 Indonesia 0.76 Lao P.D.R 0.71 West Bank and Gaza 0.44 Source: IEG desk review of ESW in 18 countries with at least two CPARs/MAPS. Note: Scores: High: 2.5–3; Satisfactory: 1.5–2.5; Moderately Satisfactory: 0.5–1.5; Low < 0.5. AAA = analytical and advisory assistance; CPAR = Country Procurement Assessment Report; ESW = economic and sector work; MAPS = methodology for assessing procurement systems. In many countries where the Bank’s work was more influential, there may have been greater country capacity (Mauritius, Morocco, the Philippines, and Turkey) (Box 3.9). Yet the Bank was also influential in Rwanda (perhaps because it could start virtually fresh, not encumbered by previous outdated laws and regulations, 46 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS coupled with concerted efforts by the government). Bank advisory work has had high or substantial impact on the first two dimensions of procurement reform—legal and institutional framework— for at least half of the public procurement systems reviewed. However, better market development and effective internal and external control frameworks have been more challenging to achieve, particularly in contexts of weak capacity and governance frameworks. The Bank also had some success in other countries in Africa, such as Ghana and Tanzania, but achieved modest results in Sierra Leone and Uganda. Although effective modernization of public procurement requires a greater focus on information, coordination, analysis, and management than on regulation, the Bank tended to focus on the implementation of rules and requirements that add processes, but not necessarily commensurate value, integrity, or transparency. There was little attention to public sector management issues such as weaknesses in cash planning, commitment control, or the absence of in-year commitment reporting. Such attention could help to better gauge and forecast the rate of budget implementation, avoiding a rush to spend at the end of the year (as described, for example, in the Ghana 2009 PEFA). Political economy issues need to be considered in interpreting scores of diagnostic exercises, as political issues affect implementation. Although the legislature may be willing to pass a law for procurement, it may not represent wide stakeholder consensus. Although the 2007 MAPS for Indonesia scores well for its macroindicator on ethics and anticorruption (BLI-12: 76 percent), the diagnostic focused on assessing the extent to which provisions were in place, and not on overall results. However, it also mentions continued weak capacity, low salaries with no satisfactory career path for government procurement practitioners, no credible independent complaints handling mechanism, and weak enforcement with no sanctions for corrupt behavior. These factors lower the relevance of the high score. Reports reveal weaknesses in contract administration, although limited attention to finding solutions. Upgrading human resources and introducing technological and information management improvements have been the most difficult to effectively achieve. The information on results offered in this analysis is also limited by the lack of quantitative information or statistical data on procurement compliance and performance. Another limitation is that the assessments in decentralized countries such as Mexico and Vietnam are typically limited to the national level. 47 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Box 3.9. Procurement Support to Turkey—Initial Positive Results of Advisory Work The Bank prepared a CPAR for Turkey in June 2001, diagnosing the status of the overall public procurement system. The government implemented the majority of the CPAR recommendations, including a new public procurement law within the next couple of years.  The Bank supported the government with a timely IDF grant for capacity building as well as by a DPL. The IDF grant provided for the development of secondary legislation, standard bidding documents and a webpage, and training programs for more than 9,000 staff from almost 300 procuring entities.  In 2005, the Bank, jointly with the government, conducted a second assessment using the MAPS baseline indicators, which was updated, largely by the government in 2007–08. Notable results reported include the financial independence of the procurement agency; the development of transparency enhancing electronic advertisements; and the collection and analysis of relevant data, validated through internal and external audits. The law had been amended 15 times; however, some amendments introduced exceptions. These results were achieved in the early years of the evaluation period, based largely on the results of the 2001 CPAR. The majority of its recommendations were implemented. In recent years, however, as IEG field visits show, the Bank has had limited engagement in the form of diagnostics, or with capacity building. It must also be noted that attribution is challenging given that the European Union, the European Bank for Reconstruction and Development, and other donors provided support. Source: IEG. PILLAR I—THE LEGAL AND REGULATORY FRAMEWORK IEG’s review suggests that although legal frameworks have been substantially or highly strengthened in all countries during the evaluation period, their implementation is more modest (Figure 3.4). Ownership of the law and its underlying principles is often limited. Agreement to legal change has been made, in the majority of cases, in consultation with the Ministry of Finance, which has led the process as a Bank counterpart. In some cases there was pressure to respond to DPL conditions. All AAA reports reviewed proposed new procurement legislation or revisions to current legislation, mostly with a view to bringing them closer to internationally accepted principles. In Albania, although a law was passed, with Bank support in addition to other factors, 15 subsequent amendments were made to comply with European Union regulations. Establishing implementation regulations was often more difficult than establishing a law, for reasons such as a conflict in regulations or the lack of a procurement manual consistent with the legislation. Yet the Bank was often able to agree to action plans to prepare documents that paved the way for implementation. 48 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Figure 3.4. Substantial Impact on Legislative Reform—Less on Action Plans Impact on Legislative Reform Action Plans to Support 13 Implementation 11 7 4 Modest High or Substantial High or Substantial Modest or Low Source: IEG. Enhancing Open Competition Results were challenging even where laws and regulations were positively influenced. Despite laws that require open competition, direct procurement often continued to be used for reasons related to the framework of implementation— weakness in the law, unavailability of funds until late in the year, and attempts by procuring entities to break procurement into small values (Box 3.10). 49 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Box 3.10. Achievements—Competitive Tendering In Tanzania, although the Public Procurement Act makes competitive tendering the main method for procurement of goods, works, and services, regulations allow the procuring entity to “select an appropriate alternative method of procurement” in “any case where tendering would not be the most economic and efficient method of procurement....” The alternatives available are single source, direct contracting (goods and works), and force account, all with specific conditions attached. The diagnostic concludes that although the approach of offering different procurement methods for different types and sizes of contracts results in an efficient system, the provision that allows the procuring officer to apply the alternatives opens an avenue of nontransparency and potential abuse and puts increased demand on the quality of the oversight authority. In Mexico, regulatory frameworks are inconsistent concerning competitive bidding requirements, the selection criteria, and the terms and conditions of final contracts. In Ethiopia and Azerbaijan, diagnostics suggest that lack of predictability in budget planning and execution left little time for open procurement. Competition in national competitive bidding has sometimes been restricted in Mexico and the Philippines, because of their strong interpretations of domestic preference. Foreign firms have been permitted to compete only when the procurement is carried out under a free trade agreement, when financed by international development institutions, or when the law otherwise permits international competitive bidding. This emphasizes the need for establishing clear criteria for trade-offs, in certain circumstances. In some countries such as Albania, although the law solidly supports open procurement, there are no data to assess the results. Source: IEG desk review of Bank procurement diagnostics. PILLAR II—INSTITUTIONAL FRAMEWORK AND MANAGEMENT CAPACITY Results of IEG’s analysis suggest that although outputs such as designation of procurement authorities and trained personnel have been achieved, there were difficulties in achieving sustainable training programs and personnel retention. The lack of adequate data in several countries has made it difficult to assess. IEG used four indicators to assess this dimension: (i) extent to which procurement is integrated into the overall public financial management system; (ii) whether there is an effectively functioning independent regulatory body; (iii) human capacity development; and (iv) adequate generation and analysis of information. Integration of Procurement into the Overall Public Financial Management System An effective procurement system functions well when it is well integrated into public financial management systems (Box 3.11). The integration of the public procurement system into the overall budget management framework was difficult; 50 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS there was substantial or high achievement in 28 percent of the countries; 55 percent achieved a modest score, and the remainder had negligible achievements. Box 3.11. Integrating Procurement into the Budget Process In Ghana, despite some positive impact, the high unpredictability of budget releases and weak commitment control negatively impacted the efficient use of procurement methods (Ghana PEFA 2009). In Azerbaijan, although there is good practice in terms of publishing an annual budget and notifications of upcoming tender opportunities, line ministries and other spenders of the budget funds do not translate allocated budgets into procurement plans. Although regulations provide for procurement planning, implementation is weak. In Mauritius, important steps have been made toward the integration of procurement in the broader public financial management systems. The government undertakes multiyear planning. The Public Procurement Act mandates that public bodies engage in procurement planning to achieve maximum value for public money and to provide for budgeting purposes. However, once again, compliance by public bodies needs significant strengthening. With the introduction of the Public Procurement Portal in April 2010, the Mauritius Public Procurement Office issued a directive that all annual procurement plans have to be posted on its website. In 2010, however, only 19 procurement plans had been posted. In Albania, the electronic government procurement platform is not linked to the budget system. This prevents automatic verification of the availability of funds for initiating a tender process. Source: IEG desk review of Bank procurement diagnostics. Establishing Functioning Procurement Authorities All reports that were the first in a sequence recommended the establishment or strengthening of a procurement authority. Such authorities have been established in all places except Mexico, but achieved without major deficiencies in only 5 of the 17 countries: the Philippines, Rwanda, Tanzania, Turkey, and Vietnam (Box 3.12). In the Philippines, an independent body for procurement is empowered by law to protect national interest in all matters related to procurement and to promulgate rules to implement the law. With greater budgetary resources it could have ensured better monitoring. In Mauritius, although significant progress has been made, the inability of the procurement agency to function independently is hampered by the higher authority of the country’s Central Tender Board. Although establishing such a functional body is relatively easy, such bodies suffer from a shortage of procurement professionals and resources (as in Albania, Ethiopia, and Ghana). According to reports reviewed, several countries may have limits to their ability to function independently in the regulation of procurement, because of political interference, lack of resources, overlapping responsibilities with other institutions, 51 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS lack of trained and knowledgeable staff, or low salaries leading to constant staff turnover. Box 3.12. Public Procurement Authority in Rwanda The Rwanda Public Procurement Authority is a public institution with a regulatory function. It has legal, administrative, and financial autonomy and is managed by a Board of Directors appointed by the Prime Minister (who appoints the Director of Procurement Authority, as well). The Board has seven members, including representatives from the private sector and civil society. The Board establishes the strategic vision and action plan, approves the internal rules and regulations, approves the annual draft budget, decides on the sanctions to be imposed to bidders and staff members contravening the law, and monitors the performance of Procurement Authority management and staff. Although there was not full separation of transaction and regulatory function, Rwandan law provided for complete phasing out of its transaction function in February 2011. Source: IEG desk review of Bank procurement diagnostics. Human Capacity Development All Bank ESW diagnosed capacity and recommended capacity development in the client countries. However, based on country diagnostics, capacity was strengthened in only two-fifths of countries reviewed, mainly in countries where there was already some level of capacity, such as Mauritius, the Philippines, or Turkey (Box 3.13). One exception was Ghana, where substantive permanent training programs of suitable quality and content for the needs of the system were established with existing institutions.55 The procurement authority has also established a help desk.56 Capacity was also developed in Vietnam and Uganda, though sustainability may be more of an issue. Lack of capacity is also a challenge for other administrative areas that are essential for effective procurement, such as internal and external audit institutions. Although the lack of any considered strategies on how this could be implemented was a factor (as mentioned in Albania, Azerbaijan, Ethiopia, Indonesia, Lao PDR, Sierra Leone, and Vietnam), the diagnostics also confirm that building procurement capacity in the absence of effective civil service reform is challenging. Ethiopia is an example where the high turnover of staff in the public sector in general was pointed out, with similar patterns in procurement-related functions. The cause is the low government salary level compared to opportunities in the private sector and in international organizations. In Lao PDR, the first CPAR notes that it was common practice for several years for government staff to take other paid work in the evenings and on weekends. The Bank report found that technical staff took time off 52 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS for parallel employment during regular working hours, often as consultants or contractors engaged on government projects and sometimes on the same project that they supervised in their government job. One consequence is delay, as government officers have less time for their regular duties. Decentralized procurement systems further strain capacity. In Albania, assessments conducted by independent authorities considered the delegation of procurement responsibilities to 1,700 procurement units inefficient and cumbersome in a country of 3.5 million people (2010 UN Report MAPS). E-procurement has helped enhance capacity in some countries. In the Philippines, PhilGEPS is a single, centralized electronic portal used by all government agencies; it also serves as a central portal providing the primary source of information on public tenders. It operates as an electronic bulletin board, a suppliers’ registry, and an electronic catalogue. In contrast, the e-procurement system established in Albania was unable to meet the transactional requirements of such a system, and it further stretched capacity. Box 3.13. Mauritius—Country Capacity and Good Governance for Effective Procurement Reform With Bank support, Mauritius achieved significant progress in its procurement reform, supported by analytical work and a timely Public Expenditure Reform Loan. However, this could not have happened without government commitment (reflected in the government vision of knowledge hub for the region) and substantial basic capacity within the country. The Public Procurement Office launched a comprehensive training program, and ministries recruited 500 purchase and supply officers. A sustainable training capacity was established at the University of Technology of Mauritius. The classes range from a certification program in public procurement and management and an undergraduate diploma in purchasing and supply management to a top-up BSc (honors) program in purchasing and supply management and an MSc in logistics and supply chain management. This demonstrates an unusually forward looking career path for procurement staff. The Public Procurement Office has also embarked on a distance-learning program through an e-learning platform, offering the flexibility that many trainees need. Source: IEG desk review of procurement diagnostics. Effective Information Systems Not Established Timely information systems, analyzed regularly for feedback and policy adjustments, are essential for an effective procurement system. Although Bank ESW has always supported development of such systems, it has not been highly 53 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS successful in this area. Recommendations were significantly followed in seven countries, but outcomes were modest or low in the remaining seven.57 Reports refer to Azerbaijan, where there are shortcomings in monitoring and complaints data and concerns about the reliability of available data. Albania established an electronic system, but with limited ability to archive or query historical information that reduced its usefulness (2010). In Mauritius, the monitoring system provides information on procurement transactions but is not integrated or automated. Information is collected manually on a quarterly basis (MAPS 2010). One exception in this regard is Morocco, which established a noteworthy information collection and dissemination system on public procurement (Morocco MAPS 2009) (www.marchespublics.gov.ma).58 Decentralization of procurement authority has constrained the collection of information in countries such as Ethiopia, Indonesia, Lao PDR, Mexico, the Philippines, and Uganda. In Ethiopia, a variable amount of information is available at local levels. In Ghana, local units are not obliged to transfer any information to the central level. In Lao PDR, a prolific amount of information is collected but remains scattered in different offices. As of the latest report in 2006, there was no tool to collect and analyze such information. In Mexico, reports noted that despite the development of the COMPRANET, an electronic procurement system, some agencies develop their own information systems, and issues of consolidation remain. In the Philippines, reports mention that existing policies require the submission of the Semestral Procurement Monitoring Report to the central unit, but with limited compliance among agencies.59 In Uganda, procurement is decentralized to more than 200 procuring entities in the central and local government, but reporting to the central regulatory body is seriously in arrears and some entities fail to report altogether. The PEFA 2009 notes difficulties in calculating how many contracts above the threshold for competitive bidding are actually competitively bid. PILLAR III—EFFICIENCY OF PROCUREMENT OPERATIONS AND MARKET PRACTICES The following component indicators were used to assess Bank support to building well-functioning procurement markets: (i) improved public procurement planning and design; (ii) increased number of timely payments or reduced delays in procurement as a result of better integration into planning and budgeting process; (iii) increased delegation of procurement authority; (iv) reduced major systemic constraints that inhibit private access to the public procurement market; and (v) improved contract administration. In all, high or substantial improvements were found in 7 of the 11 countries. Improvements in public procurement planning and design were found to be generally limited, with high or substantial results in 5 of 18 countries. According to its latest 54 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS PEFA, Rwanda reflects good practice in this area.60 Budget documentation is prepared as an integral component of the country’s medium-term expenditure framework preparation process and represents, in effect, the culmination of this process. In contrast, in the Philippines, despite regulations, procuring entities prepare the annual procurement plans in a spirit of compliance and do not take advantage of their potential as a tool to improve the procurement process and monitor its efficiency. Few agencies prepare multiyear plans that are linked to the annual budget process or to the preparation of annual procurement plans, and not all prepare or consolidate an annual procurement plan. In several countries, links between procurement plans and strategic objectives are weak (Albania, Ethiopia, Mexico [2007], the Philippines, Vietnam, and West Bank and Gaza). This affects planning and shortens the time available to issue contracts and commit funds. Agencies are then forced to adjust their initial planning to the approved budget, to use accelerated procurement processes, and to increase the use of direct contracting, which in turn decreases efficiency and effectiveness. Evidence of more timely payments or reduced delays in procurement is scarce. Not much early ESW report on this issue except in passing; therefore, the baseline is often unclear. Qualitative statements suggest limited results, with 3 of 18 countries making high or substantial progress. A recent diagnostic study for Turkey notes that “there is no major deficiency identified except that there is no provision for interest payment in case of late payments….” In the Philippines, a report noted the need to issue an obligation slip and Certificate of Availability of Funds prior to the signing of a contract by the head of the procuring entity. About 85 percent of funds are released in May, and the remaining funds are released toward the end of the year. Although there are established practices and norms for the processing of payment invoices for contracts, these are not published and there are no mandated timelines for release. Payments are generally authorized within four weeks after approval of invoices or monthly certification of progress. By contrast, in Rwanda, clear service standards have been established and payments are reportedly authorized in time. Late payments remained particularly problematic over time in countries that lack integration of procurement into the public financial management system. In Albania, this implies a lack of funds when it is time for payment. In Azerbaijan, the lack of integration into the budgetary system has resulted in untimely payments, leading to cost escalation and an increase in direct contracting. In Colombia, protracted processing of invoices and cash rationing by treasury are reported to be among the causes of delay. The Colombian government set up an integrated financial information system in 2004 that aimed to link procurement activities with 55 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS the triggering of resource commitment and payment. However, integration remains incomplete and cash availability projections are unreliable. Increased delegation of procurement authority appears to have been recommended in all cases, but not always with success. Delegation of procurement to 1,700 decentralized units adversely affected implementation in Albania. Decentralization in Uganda weakened procurement planning, resulting in emergency procurements, procurements that were not required, and procurement at higher prices than necessary. The 2010 CPAR suggests that delegation too early can lead to loss of integrity; it recommended that the public procurement authority involve itself in procurement clearances. A more gradual approach was suggested—once the capacity and control environment in procuring entities improved, the role of the public procurement authority in contract award decisions would diminish. Improved private sector access to the public contract market requires the development of market knowledge and confidence in the contracting entities that implement and administer the system. In Rwanda, government performance in advancing the private sector agenda was recognized by the Bank in its 2010 Doing Business indicators. The indicators identified Rwanda as the top reformer worldwide; it jumped 76 places in the ease of doing business, from 143 to 67. By fostering improved governance, access to credit, and streamlined regulations for the private sector, Rwanda created a favorable enabling environment for private sector participation in public procurement. Regular dialogue with the private sector also fosters better collaboration. In Ghana and Morocco, measures were taken by the government to encourage open dialogue with the private sector, formalized, in Ghana, through trade. Moreover, the annual report of the procurement authority is publicly presented, providing a discussion forum between the parties. In Mauritius, promoting private sector participation in the public sector projects was made a recognized dimension of the government’s agenda. In contrast, in Vietnam, the procurement law did not include any provision for dialogue with the private sector. Improving the capacity of the private sector to undertake procurement in compliance with the law is an important need. However, in only a handful of countries did the procurement authority train contractors to strengthen their understanding and ability to participate in procurement. In Morocco, information and training programs on public procurement were periodically offered to private sector operators. In the Philippines, partnerships to deliver procurement training were established with universities and private sector organizations, including the Asian Institute of Management. In Albania, however, the Bank reports that the 56 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS public procurement authority did not have funding to proactively engage the private sector. In Indonesia, despite the Bank’s call to strengthen the participation of the private sector, as late at 2007 less than five percent of private sector suppliers/providers participate in public sector procurement. Complexity of procurement rules further taxes the private sector. Even in Mexico, where the procurement system is relatively advanced, reports point to a lack of standardized processes and practices at the federal level, as each agency develops its own processes. This increases the cost of engagement and affects the country’s competitiveness. Contract administration, as reflected in the reports reviewed, has not been considered a part of procurement cycle in most countries (as it has been in the Bank). Accordingly, relevant information pertaining to contract management (such as change orders, revisions, cancellations, quantities of goods delivered, actual amounts paid, and so on) is not often captured. Similarly, information on contract performance and completion often falls out of the scope of the procurement authority, limiting knowledge of actual results on the ground. PILLAR IV—INTEGRITY AND TRANSPARENCY OF THE PUBLIC PROCUREMENT SYSTEM IEG used the following criteria to assess the levels of change brought out by Bank studies: (i) increased identification and sanction of corruption in procurement; (ii) strengthened independence of the dispute resolution body and increased effectiveness of the dispute resolution system; (iii) improved oversight through better functioning internal and external audit institutions; and (iv) increased accessibility to or disclosure of relevant procurement information. Only in three countries was the Bank able to make substantial change. In the other 15, Bank contributions were modest. The procurement agency in Uganda is able to identify irregularities but less able to impose sanctions. The agency is empowered to recommend action when malpractices are discovered, but it is sometimes unable to ensure that recommended changes are implemented. Its regular procurement audits revealed (end 2005) that only 2 percent of contracts audited were carried out fully in accordance with the laws and regulations. Establishing an effective dispute resolution system is as challenging as getting data on disputes resolved. Such processes help avoid lengthy judicial proceedings, and all the Bank’s advisory diagnostics have recommend such processes. But recommendations have been effectively implemented in only a handful of countries. In Mauritius, an unsatisfied bidder may request review directly by an independent 57 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS review panel. The Bank reports that panel decisions are well written and based on evidence that is on the record. The decisions, however, are not legally binding. Between 2008 and mid-2010, 41 appeals were made to such panels regarding cases handled by the Central Procurement Board; of these, 18 had merits, in the view of the review panel. Additionally, panel members do not serve full time, are often unavailable, and do not have sufficient technical support—all factors that delay resolution. Although the Bank’s analytical work has consistently recommended the establishment of complaints review bodies, they are still lacking in some of the countries reviewed. In the Philippines, a country where CPAR and MAPS have been applied more than most other countries, there was still no complaints review body as of 2012. In Azerbaijan, the 2009 CPAR notes that although an independent dispute resolution mechanism is defined in law, there is no evidence of its regular use. Actual evidence is limited, because only one case was taken to court in 2006. Given the high value of contracts processed, the limited use of the independent dispute resolution mechanism underscores its limitations. In Tanzania, interviews with stakeholders as part of the 2007 MAPS revealed that most are not aware of a complaint mechanism and that those who were commented that reviews were too protracted. In the Philippines, despite significant progress in procurement reform, the absence of an independent complaints review body has not been established to accord the appeal process credibility and efficiency. In Uganda, a follow-up Bank report notes the continued absence of an independent review body. In Ethiopia, the procurement authority has established a federal complaints board as a separate entity with the mandate to issue binding decisions. Indeed, despite Bank recommendations, some countries even lack demand for a dispute resolution mechanism. In Tanzania, despite satisfactory dispute resolution processes, there is no effective demand for them for fear of reprisal if disputes are registered. The level of awareness of the system among suppliers is low, possibly because the complaints review is not publicized on the procurement agency’s website. Procuring entities allegedly file more complaints regarding suppliers than the other way around. Complaints have increased since the establishment of the complaints board, and the decision-making process is assessed to be fair. Actual decisions are, however, not published anywhere, reducing transparency. Modern internal and external audit functions are essential, but they have not been strengthened adequately in many countries. Reorientation of the internal audit function is often hampered by the retention of traditional preaudit functions and weak focus on risk-based auditing or systemic issues. Although external audit 58 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS institutions exist in many countries, their capacity to undertake procurement audits has remained notably weak. Greater synergies between internal and external audit functions are also important. Internal audit has been more difficult to establish. Rwanda appears to be an exception, according to the Bank report (World Bank 2004). The procurement agency, the internal audit, and the external audit function have all been established and capacity developed. According to the CPAR, procurement oversight functions are being mainstreamed in all oversight institutions by building the procurement capacity of the relevant staff and the inclusion of appropriate provisions in their standard operating procedures and manuals. Concerns on coordination between different agencies remain. There is a need for interagency collaboration between procurement and corruption units. In most countries, such collaboration is limited. For example, in Ethiopia, there is no systematic exchange of intelligence and no effective tools or measures in place to enhance the efforts (that is, a red flags checklist in procurements). There are also no attempts in CPARs to better understand the entry points for corruption within the country context. E-procurement was noted in many ESW reports as reducing corruption. Following the introduction of e-procurement in Albania, the number of complaints declined from 1 in 21 tenders to 1 in 33 tenders. In the Philippines, government agencies that have adopted e-procurement have noted benefits from the transparency and competitiveness it fosters. This said, the 2012 draft CPAR led by the ADB notes that inadequate evidence is available to validate whether allegations and incidences of corruption related to procurement have been reduced as a result of the present administration’s program on good governance. Finally, timely and relevant information to monitor and manage the processes of procurement and to drive transparency is essential for effective achievement of regulatory reforms. Partnerships can be formed with civil society to provide watchdog functions, provided they have access to information. In the Philippines, the participation of the civil society organizations in the monitoring of public procurement is reported to have increased transparency of procurement, leading to economic procurement, as found in IEG’s country case study. However, their participation is still limited in scope, as well as challenged by issues of financial sustainability. There is also a lack of a clear framework for participation of specific roles, limited training of monitors, and undefined requirements for qualifications or registration processes of such civil society organizations. Many CPARs tend to 59 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS recommend that these organizations be involved in monitoring but with limited attention to the practicalities of establishing such functions. Analysis of Specific Results in Public Sector Management—The PEFA Lens This analysis of overall results in procurement reform is complimented by an evaluation using the lens of the PEFA instrument. In this section IEG examines the extent to which the Bank has used PEFAs, PERs, and overall country dialogue on public sector management issues to help its clients to achieve better integrated and more effective public financial management and procurement. The first question— the achievement of results achieved in procurement reform—is addressed through analysis of the evolution of PEFA results over time; the second question—the integration of procurement reform into better overall public financial management—is addressed through a review of country case study evidence. TRENDS IN PROCUREMENT PERFORMANCE—RESULTS BASED ON PEFA Although PEFAs can in principle include information on many themes relevant to procurement (Box 3.5), coverage is limited in practice, and only select aspects had adequate coverage for analysis. IEG assessed a sample of 10 countries (Appendix Table C.6) where there were opportunities to compare procurement-related indicators across two or more PEFA assessments.61 In the sample, around three- quarters of the procurement-related indicators could be reasonably compared for PI- 19 and eight other procurement-related indicators. Pairwise Comparison of Individual PEFA Scores Overall, the comparison between pairs of PEFA indicators that were comparable over time (Appendix Tables C.7 and C.8) suggests that around two-fifths improved, another two-fifths stayed about the same, and a fifth declined over time.62 Looking at the core procurement indicator, PI-19, the best performance was on the use of open competition, with two-thirds of sample countries improving their score and 15 percent remaining the same. 63 For the other two dimensions of PI-19—justification of less competitive procurement methods and operation of a procurement complaints mechanism—25 percent of the countries improved their score between the two PEFA assessments. Looking at the other indicators linked to procurement, the greatest improvement was in quality and timelines of in-year budget reports (PI- 24) and scope, nature, and follow-up of any external audit (PI-26). In these indicators, an average of 54 percent and 49 percent, respectively, of sample scores improved or maintained the highest score. 60 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS The greatest performance challenges related to procurement were in indicator PI-12, measuring multiyear perspectives in fiscal planning, expenditure policy, and budgeting, especially with regard to the existence of costed sector strategies. Challenges were also encountered in PI-28i—timeliness of examination of audit reports by the legislature (for reports received within the last three years) and PI- 28ii—extent of hearings on key findings undertaken by the legislature. In these cases, between 45 and 50 percent of the sample declined or maintained the lowest (D) score, between the two assessments. Grouping Patterns of Change—Formal and Functional These performance patterns were further analyzed using a methodology that categorizes each indicator/dimension by three pairs of characteristics (World Bank 2011f):  De jure and de facto changes. Some PEFA indicators/dimensions can improve their score by passing a new law or introducing a new practice, even if it is not implemented (de jure). Other indicators/dimensions require actual implementation or significant engagement (de facto). For example, under PI- 19, the dimension calling for existence and operation of a procurement complaints mechanism is de jure, whereas extent of justification for use of less competitive procurement methods is de facto.  Upstream versus downstream changes in the budget cycle. Some PEFA indicators/dimensions can earn a C or better score by altering processes early in the budget cycle (upstream), whereas other indicators/dimensions focus on points later on in the budget cycle (downstream). For example, PI- 12iii—the existence of sector strategies with multiyear costing of recurrent and investment expenditure—is upstream, whereas PI-20i—effectiveness of expenditure commitment controls—is downstream.  Concentrated versus deconcentrated changes. Indicators/dimensions under the control of central, regulatory bodies, like Ministries of Finance, are said to be concentrated, whereas those where multiple agencies or subnational authorities need to be engaged are said to be deconcentrated.64 For example, PI-24ii—timeliness of the issue of in-year reports—is concentrated, whereas PI-26ii—evidence of follow-up on audit recommendations—is deconcentrated. 61 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS All process dimensions (PI-5 and above) can be categorized under each of the three pairs of characteristics. As a general rule, de jure, upstream, and concentrated aspects comprise the formal features of a public financial management and procurement system, whereas de facto, downstream, and deconcentrated aspects comprise the functional features of a system. Analysis of PEFA scores indicates that better scores are more commonly achieved for the formal features than for the functional features. The presumed explanation is that formal progress can be achieved through mainly technical means, focused on a few agencies and at a high level in terms of process. Formal progress mainly takes the form of laws and formal rules that need to be in place before functional improvements can occur. Functional process reform is more difficult to achieve because it is more difficult to coordinate the work of many agencies, and because reforms actually implemented may threaten rents and face greater political and bureaucratic opposition. An analysis of 33 countries with repeat PEFA assessments found evidence that change in public financial management was more likely to occur in areas that can be readily monitored, that didn’t threaten vested interests and patterns of rent seeking, and that could be implemented by a single organization. 65 The analysis found that over time, formal features would be more likely to maintain top scores or improve, and functional features to maintain low scores or decline (World Bank 2011c). Evidence from IEG’s sample countries is broadly consistent with this analysis, with formal features mostly doing better than functional features (on the right in Table 3.10).66 Table 3.10. Best and Worst Scores by Dimension Dimension type % of scores Dimension type % of scores Highest or improving scores Upstream 32 Downstream 32 De jure 37 De facto 30 Concentrated 42 Deconcentrated 29 Lowest or declining scores Upstream 22 Downstream 17 De Jure 6 De facto 22 Concentrated 10 Deconcentrated 21 Source: IEG analysis. These findings suggest that greater effort is needed to make progress in the more difficult aspects.67 Yet reforms can be costly and politically risky and should focus on solving the highest priority, locally identified problems. 62 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS BUILDING INTEGRATED CAPACITY FOR PROCUREMENT AND PUBLIC SECTOR MANAGEMENT IEG reviewed the extent to which both PEFA assessments and PERs supported by the Bank contributed to procurement performance improvement, as well as to the integration of procurement within a wider framework of public financial management. Because the contributions of Bank inputs, and especially the PEFAs, were only one of many factors influencing outcomes, it is not possible to prove attribution. However, there are indications of the extent of influence.68 Influence on Country Policies toward Public Financial Management and Procurement In principle, PEFA assessments can be useful for bringing together Poverty Reduction and Economic Management (PREM), financial management, and procurement staff to focus on the procurement reform agenda. First, looking at the extent to which PEFAs have contributed to broad improvements specifically in procurement performance, there is little evidence that can be mainly linked to PEFAs or PERs. Rather, more comprehensive diagnostics such as CPARs and MAPs have been more important. For example, for Ethiopia, IEG’s case study indicated that the improvement in country procurement systems over the past 10 years can be attributed to the Bank’s analytic work (CPARs) and support for their implementation The PEFA analysis draws on the CPAR but did not initiate further analysis on procurement-related issues. Similarly, the State Procurement Authority in Azerbaijan stated that support embedded in investment lending was the most successful in terms of capacity building. The Authority also appreciated that the CPAR/MAPS provided a roadmap to reform but was unaware of the PEFA. In Mexico, findings from the CPAR helped to provide the basis for procurement reform, and some of this procurement analysis was repeated in the PER.69 Tanzania showed solid improvement on PI-19 between the 2006 and 2010, but the PEFA exercise was not highly regarded by the government, and it is likely that there were other factors leading to the improvements. In Peru, the PEFA assessment was ongoing at the time of a MAPS diagnostic, which served to inform the PEFA; this may have helped shape new laws on government procurement and central purchasing/shopping (European Union, World Bank, and Inter-American Development Bank 2009). However, there is some evidence that PEFAs may have contributed to performance improvements in public financial management areas that are indirectly related to procurement (see Appendix 3). 63 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Influence on Country Integration of Procurement and Public Financial Management Although PEFA assessments may not have helped secure major elements of procurement reform or the capacity building, and in practice seemed often to have relied on ongoing procurement assessments to inform indicator PI-19, there is evidence that Bank client countries have increasingly approached the issues of public financial management and procurement in a more integrated way. In some countries, the Bank has contributed to this process through its collective instruments (Box 3.14). Box 3.14. Building Integrated Capacity—Public Sector Management and Procurement In Bangladesh, since a procurement law was adopted in 2006, related reforms were introduced in other key parts of the budget cycle, including the adoption of more strategic budgeting through a medium-term budget framework and the passage of the Public Money and Budget Management Act in 2008. This latter consolidated a government budget policy framework, computerization of accounts, and some improvement in the conduct of audits. The Bank was critically important in pushing through these reforms through a range of instruments, including the 2002 Country Procurement Assessment, NLTAs, DPLs, and investment loans. Bank staff claim that the PREM, procurement, and financial management functions in Bangladesh are well integrated. A recent $95 million trust fund that has supported reform in the Ministry of Finance, Parliament, and the Office of the Controller and Auditor General had a dedicated subcomponent on the integration of the procurement plan of the government with the budget. The results of the corresponding analytical work are summarized in PERs and in the 2011 PEFA assessment, so in that sense these instruments played a small role as a diagnostic instrument in motivating the reforms. However, as with other cases, their main role is in setting out detailed justification for improvements in other parts of the budget cycle that are linked to procurement. In Indonesia discussions are ongoing to better integrate procurement with budget and expenditure management in two areas: linking procurement planning to the budget process and linking procurement outcomes to asset management and logistics. Lembaga Kebijakan Pengadaan Barang/Jasa Pemerintah is currently developing the Procurement Management Information System. Two modules—vehicles and pharmaceuticals—are complete (see http://www.lkpp.go.id). It is planned to link the system to budget, planning, electronic procurement, and monitoring and evaluation functions. In Morocco the PEFA makes reference to linkages between investment budgets and forward expenditure estimates, pointing out that recurrent cost implications of investment decisions are included in forward budget estimates only in a few (but major) cases. The PEFA also mentions the progress and the increasing number of independent audits being undertaken that relate in particular to procurement 64 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS procedures. The requirement to address public procurement and financial management reforms in parallel is mentioned in the 2009 Country Partnership Strategy as part of the general objective to support the public administration reform, including the improvement of the public expenditure management system. In Peru recent procurement diagnostics have substantively discussed the integration of procurement into the budgeting, planning, and audit process. This has been noted in the Country Partnership Strategy. The integration of procurement into the budgeting, planning, and audit process was not included in the recent 2009 PEFA. Recent public financial management, PEFA, or PERs do not discuss (i) whether procurement planning and associated expenditures are part of the budget formulation process and contribute to multiyear planning; (ii) whether the budget law and financial procedures support timely procurement, contract execution, and payment; and (iii) whether procurement actions can only be initiated with existing budget appropriations. In the Philippines PREM has supported the Government Integrated Financial Management Information System. This involves the Departments of Finance, Procurement and Budget Management, and Audit. The roadmap recognizes the integration of procurement into the budgeting and expenditure management functions. The CPAR has indicators that touch on financial management, for example, audit, budget, and budget planning. Financial management and the CPAR (procurement) currently come together in Bank country operations at the CAS. In Senegal government policy regarding public finance modernization has been based on the 2003 Country Financial Accountability Assessment diagnosis and the 2003 CPAR. The reform process started with the approval of action plans proposed in those reports and is coordinated by the Project for Budgetary and Financial Reforms Coordination, monitored by a project unit within the Ministry of Finance. The project has supervised the reforms of both the public procurement system in 2007 and the public finance system. Progress is measured in the 2007 and 2011 PEFAs. Ex ante control of the existence of budgetary resources and obligations related to the preparation and publication of annual procurement plans are among the new provisions of the 2007 reform of public procurement. The 2011 PEFA also provides for integration of the Public Finance Management Integrated System with the public contracts integrated management system. The connection of the two systems, planned since 2009 to increase the integration of the public procurement into the national public financial management system, should be implemented soon. Source: IEG case studies. Influence on Stakeholders Outside the Public Sector The publication of reports on the PEFA website, and in some cases on government websites, which has occurred in all sample countries, can strengthen transparency in budget processes, thereby also helping procurement. The posted indicators of countries sampled here are Albania, Bangladesh (only indicator ratings for baseline assessment), Ethiopia, Ghana (federal government only, not the subnational 65 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS assessment), Honduras, and Indonesia (baseline assessments only), Moldova, Mozambique, Senegal, Serbia, Sierra Leone, Tanzania (five central government assessments, including one as part of Public Expenditure and Financial Accountability Review), and Timor Leste. Evidence from Country Case Studies VIEWS IN THE FIELD – QUALITY OF DIAGNOSTIC INSTRUMENTS In conclusion, IEG reviewed evidence of the Bank’s support for capacity building through its advisory work, collected over the course of its 11 field visits, which covered countries in each Bank region. Although evidence from desk reviews necessarily stops at the time of the latest available report on any given country, IEG’s field visits (October–December 2012) describe the current situation and offer perspectives on diverse stakeholders’ views of Bank contributions. Information from field visits is supplemented in subsequent sections by in-depth desk reviews of the role of procurement capacity building in country strategies, in analytical and advisory work, and in Bank lending operations. An overall perspective requires triangulation of information from each source. Capacity building in procurement is broadly defined to include the building of legal systems for procurement, strong institutions with capable staff, efficient markets, and provisions for transparency and integrity in procurement. Overall, findings that emerge are similar to those from the desk reviews. The Bank made substantial efforts to build procurement systems starting in the late 1990s/early 2000s in almost all the countries visited. These efforts consisted, in the first instance, in undertaking CPARs to review the legal and institutional framework for procurement in country and to recommend reforms. The first generation of CPARs consisted of mandated instruments, which are seen as comprehensive, thorough analyses, much appreciated in country; they define the roadmap for procurement reform going forward (Box 3.15). They also provided an effective basis for policy dialogue with country counterparts. Box 3.15. CPARs: Roadmaps for Procurement Reform—Especially Legal and Institutional In Ethiopia, two CPARs were undertaken over the review period—in 2002 and in 2011-12. There is substantial government commitment to the procurement reform agenda, and the Bank’s analytic work had strong ownership in the case of both reports. The first CPAR led to a number of actions, including the proclamation of the Procurement Law, establishment of an independent procurement oversight and regulatory agency at the federal level, and the establishment of an agency responsible for entering into framework contracts. Implementation support for the first CPAR was provided by procurement-focused 66 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS components of two Bank projects. The 2012 CPAR identified additional actions; phased implementation of its recommendations will be supported by elements of a multisector project for basic services (Protecting Basic Services 3). In Bangladesh, the 2002 CPAR was the foundation document for all subsequent Bank interventions in procurement, including reforms supported through policy-based lending and its two investment projects. Although not formally updated, Bank office involvement in country procurement matters remains extensive, tracking of progress is active and ongoing, and there is a high level of up-to-date knowledge and awareness of the state of play. The Bank also carried out a Country Financial Accountability Assessment and has been leading the donor group's dialogue with the government on procurement reform. In Tanzania, the CPAR was also cited as an important piece of analytic work for the country, led by the Bank and supported by a high-level task force comprising representatives from ministries, local government, parastatals, and the private sector. The CPAR provided a detailed analysis of the existing situation in the country and recommended a series of actions, including updating the procurement law, establishing a public procurement authority, building capacity, addressing outreach needs, supporting audit and anticorruption measures and public sector management, monitoring contracts, and setting up a complaints mechanism. Follow-up actions included enactment of a public procurement act in 2004, preparation of associated regulations in 2005, a new procurement related law that came into force in 2011, and the associated regulations, expected to be gazetted in 2013. In Mexico, the 2007 CPAR was an important analytical instrument, instrumental in helping Mexico’s procurement reform. Its recommendations were useful for reforms of the legal, institutional, and operational aspects of the procurement system. In Turkey, a CPAR was completed in 2001, which served as the foundation for a range of reforms. And the 2001 CPAR of Indonesia was also cited as a highly relevant guide to needed reforms. Source: IEG field visits. Once CPARs ceased to be mandated, and because of the use of MAPS for the Bank’s use of country systems experiment, their frequency declined. Although MAPS exercises on their own were perceived to have limitations, new hybrid CPAR/MAPS exercises undertaken in some countries (Ethiopia and the Philippines) were acknowledged as the most useful (Box 3.16). Yet the frequency of such exercises has clearly declined, and as a result, the Bank’s procurement work is perceived to have lost some depth and traction over time in some major borrower countries. The implementation of recommendations to emerge from Bank diagnostics was supported in many case study countries by a variety of Bank lending and grants. An assessment of their relative contributions is discussed in this report (Chapter 4). The present section focuses on the evidence regarding overall results achieved in terms of the implementation of the recommendation of Bank diagnostic work. 67 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS Box 3.16. Early MAPS Exercises: Deemed to be Less Useful Limited Early Support for MAPS  In Ethiopia, the government did not find the MAPS exercise useful; hence, a second CPAR was undertaken in 2012, using the MAPS methodology for diagnostics but going well beyond to develop recommendations and an action plan. Ethiopia’s 2012 CPAR covered all areas in depth and contained "mini CPARs" of the 11 regional states or city administrations.  In Azerbaijan, a CPAR was completed in 2008. The State Procurement Authority enjoyed substantial participation in this exercise. The MAPS was perceived as a less comprehensive evaluation of the country system, viewed largely as a checklist to provide a grade on country procurement capacity and not as a comprehensive evaluation or road map to reform.  In Bangladesh, an OECD-MAPS baseline-only study was undertaken in 2006 as part of a public expenditure and institutional review and was published as an annex to the review. This latter exercise gave some useful information according to respondents in country, but because the report reflects information at a point in time, it is of limited application when detailed, up-to-date information is needed. Partial Endorsements of MAPS, Properly Used  In Indonesia, the MAPS is seen as a tool for a CPAR-like process, not as a replacement for it. The central procurement authority mentioned two MAPS exercises, in 2006 and 2008, that the government undertook. The latter included consultation with all the development partners, including the Bank, but no report was issued. As others have noted, the procurement authority found the CPI of the MAPS to be inadequate. In 2011 it conducted a pilot survey of compliance using the CPI, but significantly expanded and modified it to meet the survey objectives. With such modifications, country procurement staff reported that MAPS has been useful as a more structured instrument, providing a greater degree of government ownership.  In Mexico, some recommendations of the CPAR were reached based on MAPS indicators. The MAPS, conducted in the context of the use of country systems pilot between 2008 and 2010, is also intended to be used as a road map for reform, beginning in 2012–13.  In Peru, respondents mentioned several analytic products, including the MAPS of 2009, the PEFA of 2012, and the CPS of 2011, including a policy note on procurement, that together provide a good baseline for knowledge of the current system.  In Morocco the first CPAR was conducted in 1999, and a later decree on public procurement (2007) incorporated some of its recommendations (in particular reinforcement of ex post audits and reduction of ex ante controls). A “mini” CPAR, directed toward procurement by the Ministry of Education, was undertaken in 2007. The dialogue initiated with Moroccan authorities in the first CPAR has been maintained and has been reinforced during its subsequent MAPS evaluations, undertaken for the pilot. It is intended to provide a roadmap for a revised public procurement code and will also be a condition for disbursement under a new DPO.  In Senegal, the CPAR was the sole piece of analytic work highlighted by country 68 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS respondents in the procurement area. Conducted in 2003 jointly with the African Development Bank, it raised substantial issues in relation to transparency and efficiency. Its action plan was approved by the government in 2005, after consultations with private sector and civil society, and a revised procurement code was adopted in 2007. In addition, later MAPS exercises (2008) served as a basis for the Bank’s use of country systems assessment. Together they are deemed today to provide in-depth knowledge of the Senegalese procurement system. Both were carried out in close collaboration with the government. Source: IEG field visits. VIEWS FROM THE FIELD: RESULTS In most countries, at least some actions were taken based on the recommendations of Bank diagnostic work. Yet in all, there was an absence of a comprehensive procurement capacity-building strategy supported by an action plan. Bank support was considerably focused in the areas of legislative and regulatory reform, as well as in building procurement institutions. Some attention was, however, given to developing better modalities and markets. Human capacity development was attempted, though mostly with a focus on immediate needs for implementing Bank lending. The Bank’s Advisory Work—Results in Legal and Regulatory Reform In 10 of 11 countries, Bank procurement staff and country clients deemed legal reform as a substantial part of country procurement support programs. Further, Bank procurement staff and country clients share the view that, in most countries, the Bank has been effective in terms of helping countries align their procurement laws with best practice—Bank procurement staff deemed this help substantial or high for 7 of 11 countries; country clients were even more approving, with a high or substantial rating in 8 of 10 countries. Yet results have been mixed. Although there was undoubted progress in many instances, there were also cases of stalled enthusiasm for reforms or even reversals, underlining the long-term challenge of change (Box 3.17). Box 3.17. Results of Advisory Work—Legal and Regulatory Reform: Evidence from Field Visits In Azerbaijan, the 2008 CPAR recommendations were encapsulated in a 2012 open government decree, and the procurement authority indicates that it should now be implemented. Management reports that the Bank has supported the Procurement Authority in the review of draft amendments to the Public Procurement Law, complaint handling mechanism, and debarment procedure. 69 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS In Bangladesh, stakeholders agree that the Bank and the ADB have been the main instigators of procurement and legal reform, which has included the promulgation of the Public Procurement Regulations (2003), Procurement Procedures (2004), Rules (2008), Public Procurement Act (2006), as well as operational tools, particularly the standard bidding documents. The Bank also provided cabinet advice on the e-signatures law. The Bank also contributed to the creation of the Central Procurement Technical Unit, an institution that has become the lead actor in most procurement reform, including the development of the electronic procurement system, and is directly involved in all centrally arranged capacity building in procurement. However, Bank staff also point out recent modifications to the law, passed by the government, which has some differences from the Bank’s model. Substantial results were achieved in Ethiopia as a result of the Bank’s ESW work. A national procurement agency was established, a procurement proclamation was issued, and procurement agencies were established at the state (province) level. The Bank also assisted the government in drafting standard bidding documents and procurement manuals. In Indonesia, the CPAR and MAPS aimed at legal objectives (passage of a procurement law), regulatory reform (setting up the central procurement authority), and human resources development (procurement professionalization). Success has been mixed. The establishment of the procurement authority was an important achievement, but a suitable law was not passed, and the current law is seen as deficient in many respects. Diagnostics are somewhat old, and recent interventions (apart from a DPL) have been limited. In Peru, results envisaged in knowledge and advisory work have been achieved to a modest extent. The law of 2008 included the establishment of a government procurement center to deal with corporate procurement at the level of the national government. This has not been implemented to date. Peru was the only country in which Bank procurement staff and country clients deemed legal reform to be a negligible or modest part of the country procurement support program. The reform of the legislation was found to be largely country led and the new law remains restrictive. Informal Bank advisory support appears to have been more important, with some contributions to the ongoing reform through training programs, capacity building, e-procurement advances and meetings to discuss overall strategy and the results of the CPAR. The Bank was also a key player in a recently signed cooperation agreement on procurement, between Chile and Peru although attribution to Bank ESW/AAA is difficult. Bank analysis conducted in the context of the preparation of the 2012-16 CPS included a policy note on procurement that details issues and proposes medium-term reforms. In the Philippines, the Bank played a lead role in developing the procurement law, also involving the government and other development partners, which is considered “good, although not perfect” by procurement staff. The implementing rules and regulations were highlighted as good practice, also developed with Bank assistance. And the procurement manual, developed with Bank assistance, provides national standard bidding documents and has been a successful tool in developing sound procurement practices in different agencies and different levels of government. In Senegal, at the conclusion of the 2003 CPAR the action plan proposed was approved by the government in 2005. Drafts laws and regulations were prepared with a technical assistance financed by the Bank that laid out general principles related to public contracts, 70 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS including those applicable to procurement. An oversight authority for public procurement was established, handling complaints from bidders, carrying out independent audits, and designing national capacity development. As a result, the regulatory and institutional framework and capacity management of public procurement in Senegal improved considerably from 2005. However, it seems that the government’s willingness to pursue reforms in the direction suggested by the reports has decreased. The action plan has not been followed and several decisions and decrees and the modification of the procurement code enacted in 2011—including a new modification in its scope in January 2012—led to less transparency and reliability in procurement procedures. In Tanzania, significant progress was achieved as a result of its CPAR, especially in legal and regulatory reforms, institutional strengthening, capacity building, and controlling fraud and corruption. A new law was enacted in November 2011. Bank staff were actively involved during the reform process, supporting the preparation of the law and directly and indirectly supporting capacity building efforts. Procurement processing time has reduced and efficiency improvements can be seen. a. Source: IEG field visits. VIEWS FROM THE FIELD: HUMAN CAPACITY DEVELOPMENT Field visits illustrate Bank support for procurement staff capacity building in several countries, typically through hands-on support to implementing agencies (Box 3.18). Few programs looked at longer-term sustainability issues, though some governments made efforts to support sustainability by developing career paths for procurement professionals, and courses at local universities or institutes of management, and by trying to stem ever-present attrition by training new entrants. Thus, reasons for poor performance in procurement capacity building arise from endemic country issues as well as Bank-related issues. Government resources are stretched, civil service salaries are low, and staff rotation or movement to positions where their newly acquired skills are not needed is typical. Building procurement capacity is an integral part of wider civil service capacity building and is hard to develop in isolation of the overall civil servant cadre. The lack of recognition of procurement as a professional career path is an impediment that some governments have tried to address. At the Bank, programs have been focused first on implementing Bank projects. Country government staff as well as Bank procurement staff sometimes questioned the extent to which routine procurement training in the Bank includes “cutting- edge” expertise. Bank staff in the Philippines asked whether the Bank has the expertise to advise client countries. The Bank’s human resources policies may adversely affect procurement capacity development in client countries, as the policy of hiring extended term consultants for a maximum of two years means the loss of 71 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS scarce procurement skill and the need to train a new set of incumbents who would also leave in due course. Time-consuming and protracted Bank hiring processes have also been mentioned. Few country departments explicitly provide for training- related resources, as separate from transactions-related work. Ethiopia’s work program agreement is one exception. Box 3.18. Results—Building Human Capacity In Bangladesh, focused support for building procurement management capacity was provided by establishing a regular training program. It included provisions to train the trainers and to provide incentive mechanisms for sustainability, through accreditation and higher-level degrees in conjunction with overseas institutes (Box 4.2). However, even in Bangladesh, the degree to which capacity is improving is qualified: training numbers are reasonable, but trainees have little prior knowledge and results indicate a large proportion of poor grades. Training is compliance based and does not provide exposure to alternative models or complex procurement. Moreover, the long-term financial stability of the training programs was in some doubt at the time of the IEG visit (November 2012). In Ethiopia, the role of the Bank in training and procurement capacity building is highly regarded, though a focused is on the implementation of Bank projects. Capacity building is provided through hands-on support and training/workshops, for example, short training programs, procurement clinics, briefing sessions, and regular programs with the Ethiopian Management Institute to conduct training on the Bank’s procurement guidelines and procedures. Ministry officials and implementing agency staff also participate in such programs. Training programs are now being provided on a cost-recovery basis, so the initiative is sustainable for the Institute. Although a procurement cadre is being formed, there is considerable movement of such trained staff to the private sector. The national procurement agency remains committed to the continued professionalization of procurement staff at all levels, and its proactive efforts have helped reduce net attrition through training of new entrants. In the long run, the government is aiming for a career structure and career path for procurement experts. A competency framework has been developed, and there is an initiative to develop university- level procurement “certificates,” diplomas, and a degree course at the master’s level at the Ethiopia Civil Service University. This and other Bank support for the professionalization of procurement staff has been important. The second CPAR contains a broad-based action plan for further reforms that will be supported through the Bank’s recently approved Protection of Basic Services (Phase III). The Bank has also played a substantial role in training and capacity building in Tanzania. At the level of individual projects, the Bank provides frequent training to implementing agencies through workshops and clinics, as well as a regular annual program. More than anything else, the Bank’s advisory role – as noted in comments and feedback on contract documents -- is highly valued. The Bank is considered responsive and generous in its advice, and field office staff are highly regarded. In Peru, an IDF grant has been prepared to build capacity at the national level, starting in 2013. The objective is to support the design and implementation of a sustainable training 72 CHAPTER 3 SUPPORTING CAPACITY THROUGH AAA—CPARS, MAPS, AND PEFA TOOLS program on international best practices and innovative procurement methods, particularly for infrastructure projects. Trainees will serve as a resource group to disseminate skills nationwide– this is significant in a country with 1,800+ procuring entities and limited procurement capacity in remote areas. The Philippines offers a good example of the Bank’s efforts to institutionalize capacity building through the Asian Institute of Management. The Bank and JICA worked with the Asian Institute of Management in the late 1990s to develop a curriculum, and the program is accredited. The program has resulted in a cadre of trained government procurement specialists, available to project implementing units and other agencies involved in procurement for Bank-financed and non-Bank-financed projects. Those receiving training must agree to stay for a minimum period of time. There is currently no certification system for government procurement specialists. Source: IEG field visits. 73 4. Supporting Procurement Capacity through Bank Lending Lending Operations That Supported Building Procurement Capacity In many countries, the implementation of the recommendations of the Bank’s diagnostic and advisory work was supported by an assortment of lending, grant instruments, and hands-on support.70 Apart from investment lending and technical assistance, support for strengthening national procurement systems has also been offered through the Bank’s policy-based lending operations (DPOs), especially those with a focus on fiscal management or public sector reform that sometimes include procurement reform issues in their scope of policy reform. In numeric terms, support through DPOs has been frequent and far more significant than support through investment lending—perhaps also reflecting borrower reluctance to borrow for technical assistance when grant funds, even if limited and uncertain, were also available. Over the past decade, around 200 have had prior actions that refer explicitly to procurement-related policy reforms; around 40 had at least 15 percent of total conditionality in this area, in terms of numbers of conditions.71 A core vehicle for procurement strengthening in the Bank was the programmatic DPOs known as PRSCs. These were introduced in 2004, with a specific focus on better public sector management (IEG 2010e). Particularly in Africa, many countries had several series of PRSCs, many of which supported strengthened procurement (Box 4.1). Apart from PRSCs, the Bank also made use of other DPL series intended to support fiscal management, or institutional development, for procurement support. Main Findings and Recommendations  The Bank has supported the implementation of its diagnostic and advisory work in procurement reform through a substantial number of DPOs and IDF grants, but with scarce technical assistance–focused investment lending. Policy-based lending was a preferred approach, as it was seen to be a successful instrument for taking the procurement reform agenda forward.  Bank support addressed a substantial range of issues, albeit with some concentration, especially in DPOs, on the legal framework as well as institutional strengthening. The policy-based lending instrument by its nature is less well suited for building institutional capacity, which cannot be promoted simply by means of “triggers.” 74 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING  DPOs paid less attention to efficiency issues or to more advanced procurement market strengthening or longer-term capacity development. Technical assistance investment lending provided more hands-on support in these areas, although in rare cases. Only two countries received technical assistance loans for dedicated support to procurement reform.  Capacity-building and market development issues were also tackled by IDFs, often with some success. Because of their small size and uncertain nature, these were a less reliable vehicle for the operationalization of procurement reform.  The main source of ongoing support for building procurement systems and capacity in country was by means of within-project support to implementing agencies, to assist them in undertaking project-specific procurement in the context of individual investment projects.  Bank support addressed a substantial range of issues, albeit with some concentration, especially in DPOs, on the legal framework as well as institutional strengthening.  There was little focus in DPOs, except in a mechanistic way, on the integration of public procurement into the wider framework of public financial management  Many countries had mutually interlocking patterns of support for procurement reform through a combination of DPOs, IDF grants, and investment lending. This appears to have been beneficial.  There is limited scope for changes from a single lending operation—engagement over time is essential and the pace of reform is slow. There is value in undertaking reform through programmatic, longer-term, or repeat instruments.  In some cases, even with legal and institutional reinforcement, there is little sense of change in the implementation of procurement practices.  Going forward, the Bank may need to move from its present substantial focus on the DPL vehicle toward more hands-on support through technical assistance. As client countries’ procurement systems mature and basic legal and institutional structures are put in place, there may be less scope for achieving change through policy-based loans.  Given the relatively small size of IDF grants, technical assistance loans may provide more comprehensive, reliable, and longer-term support. Such loans will have to be built in to country department work programs. The Bank also made a limited number of technical assistance investment loans that focused on strengthening borrower countries’ procurement systems (Box 4.2). Although IEG’s first screening identified 193 investment projects referring to public expenditure, financial management, and procurement, a further filter based on the nature of activities found only around half of the projects (101) had any procurement-specific elements, and only 14 had somewhat high procurement content. Only six operations had a primary focus on strengthening procurement.72 75 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING Box 4.1. Supporting Procurement Capacity through PRSC Series Mozambique had five DPLs that dealt with procurement; all were part of the programmatic PRSC series that included, in all, seven operations in three series that span from 2005 to 2011. During the evaluation period, two investment loans were approved: the Public Sector Reform Project in 2003 and the National Decentralized Planning and Finance Project in 2010. Procurement constituted around one-eighth of the content of total prior actions and was aimed at creating and strengthening procurement regulations and institutions:  Passing the procurement code was a trigger for PRSC I in 2004; it was not met and became a first tranche condition for PRSC II. The code was approved by the end of 2005.  In PRSC II, second tranche conditions included the approval of a revised implementation action plan, carrying out procurement audits in at least two ministries, preparing training programs for civil servants and suppliers, and preparing terms of reference for elaboration of standard bidding documents.  PRSCs III and IV supported the implementation of the new procurement system at the district level through staffing and issuing standard bidding documents.  PRSC V continued with a third phase of support at the district level.  PRSC VI supported improvements in the number of contracts of the public sector subject to public tender (at least 88 percent), increased reporting of noncompetitive contracts, and an operational complaints process.  As indicated in the PRSC VI Project Appraisal Document, Mozambique has made progress in the area of developing the legal and regulatory framework for public procurement since the approval of its decree 54/05 and the creation of its regulatory body in 2006.  The ICR Review for PRSC V indicates that 1,860 technicians have been trained at central, provincial, district, and local municipalities levels to facilitate the adoption of the new procurement system. Mozambique has moved into a system based on open competition as the default setting; however, better monitoring and evaluation systems and effective complaint mechanisms are needed. Five of Tanzania’s eight PRSCs had procurement-related prior actions. The Tanzania Accountability, Transparency, and Integrity Project of 2006 and the Strategic Cities Project of 2010 also supported procurement reform.  Second PRSC: Preparation and submission to parliament of a draft bill amending the 2001 Public Procurement Act.  Third PRSC: Establishment of a regulatory authority for procurement; decentralizing procurement to procurement authorities.  Fourth PRSC: Approval of an organization structure for the Public Procurement Regulatory Authority, and allocation of additional budgetary resources to it.  Eighth PRSC: Presentation to Parliament of amendments to Public Procurement Act to increase autonomy of the Public Procurement Regulatory Authority, power to enforce procurement rule. According to the ICR for PRSCs IV–VIII, the power and autonomy of the procurement agency to handle breach of procurement process was enhanced. The organization of the Public Procurement Regulatory Authority was approved and appropriate budgetary resources allocated. New procurement regulations were prepared but have not yet been gazetted. The new Public Procurement Act that increases 76 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING the autonomy of the Public Procurement Regulatory Authority and its power to enforce procurement rule was passed by the Parliament. The percentage of procurement entities complying with the Public Procurement Act was 73, falling somewhat short of the 80 percent target. Source: IEG desk review. Box 4.2. Investment Lending/Technical Assistance Projects with a Procurement Focus Only two countries—Bangladesh and Bhutana—received technical assistance loans with procurement as the primary focus. Bangladesh’s two loans—Public Procurement Reform Project I and II—were approved in FY02 for $4.5 million and in FY08 for $23.6 million. The latter is under implementation. Bangladesh’s first loan was aimed at establishing a central procurement technical unit; implementing procurement reforms and rules and procedures, and improving procurement management capacity.  Legal and regulatory framework: The project supported drafting of the procurement law, rules and procedures, and standard bidding documents (including models for community contracting).  Institutional framework: Establishing a central procurement technical unit, recommended by the CPAR. Support was also given for a computerized procurement management information system.  Increased Transparency and Accountability: A public procurement website was launched, and provisions were made for contracts to be published and better advertised. An appeals procedure, procurement audits, and a code of conduct for procurement professionals were also undertaken.  Capacity Development: The project aimed at providing local training for 1,600 government staff and building a critical mass of 25 trainers to sustain in-country training capacity. Bangladesh’s second investment loan for support to procurement focused on developing a more extensive training program and strengthening procurement management in sector agencies. It also contributed to areas of—  Legal and regulatory framework: Ensuring that all secondary legislation affected by the adoption of the law would be amended as needed.  Institutional framework: Improved monitoring and management of procurement processes in targeted agencies.  Building well-functioning procurement markets: Practical assistance with issuing further standard bidding documents, and increasing the functionality of e-procurement through pilots.  Increasing transparency: Social awareness of the procurement process through the development of a communication strategy and an advocacy campaign.  Capacity development: The focus of this project was to develop a more extensive training program, institutionalizing it. The content of the training would enhance core procurement skills, not fully developed in previous project, such as evaluation 77 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING mechanisms, contract administration, and planning, budgeting, and supply chain management skills. The project will provide training of a procurement focal person in targeted agencies. Incentive mechanisms would be introduced, including recognition, access to accreditation programs and certification. Bangladesh also had other parallel lending that reinforced procurement reform, including the series of Development Support Credits (I–IV). The overall rating of the first technical assistance loan, according to its ICR Review, was Moderately Satisfactory, partly because of limited actual evidence of results. The ICR Review pointed to risks in terms of political support, weak capacity of Central Procurement Technical Unit to implement reforms and broader public sector inefficiency. According to the latest ISR (June 2012) of the subsequent operation, staffing of the Central Procurement Technical Unit is a challenge while piloting of e-procurement was ready to be rolled out. Progress was reported despite changes in the Procurement Act that are inconsistent with World Bank guidelines in some areas. Source: IEG. a. In Bhutan, one technical assistance loan was approved in FY08 for $1.5 million. More numerous were a series of trust-funded IDF grants that focused on procurement reform or on overall public sector management, including specific elements related to procurement. More than 200 IDF grants refer to public expenditure management–related themes, and around 62 over the last decade bear titles that refer explicitly to procurement (see Appendix B). Box 4.3. Institutional Development Funds Support for Procurement Reform In the Philippines, six capacity building grants (five IDF and one Japan Social Development Fund grant) contributed, over the last decade, to building procurement capacity. The average allocation for the IDFs was $250,000. The Philippines did not have any other parallel lending for procurement. The roadmap for guiding the work was outlined in the CPARs, and coordinated with other development partners. Areas supported included:  2002: Creation of Government Procurement Policy Board, technical assistance for preparing an implementation plan, preparation of standard bidding documents, the development and early implementation of a training program, and a public awareness campaign.  2005: Strengthening the audit function and preparing a quality assurance program for internal audit units; creating a national training program and certification for auditors. Regarding procurement, it supported the implementation of a procurement records management system.  2007: Further support for the creation of a career stream for procurement practitioners and the implementation of public procurement training modules and certification.  2008: Support for the implementation of an e-procurement system –including training to assist users—that is, procuring entities and suppliers. Technical assistance to the Procurement Service Department of Budget and Management for managing and 78 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING reviewing the system. As reported in the ICR memoranda, the Procurement Records Management and Monitoring System for national agencies was finalized and accepted by the Government Procurement Policy Board. Eighteen trainers were trained and are capable of providing training to the rest of public procurement practitioners. A generally satisfactory training program and modules were delivered. A report covering the detailed steps to create a career path for public procurement positions in the bureaucracy has been produced. Despite delays, some agencies are already requiring e-bid submission in some of their smaller procurement activities. Training on e-procurement was provided; as a result, certified trainers were identified: 157 from different agencies, 48 from the Procurement Service, and 14 from the e- procurement agency. In Morocco, IDF grants have been directed to support public procurement reforms and human capacity building. The 2006–09 IDF grant was aiming at (i) establishing an electronic procurement information system, which is now used throughout the country; (ii) designing and conducting a high-level training program for 40 trainers; and (iii) implementing a new public procurement system throughout the whole Moroccan administration. A new IDF grant (2010–13) aims to realize a diagnosis of training needs and the definition of a sustainable national training strategy for procurement specialists and assisting the Moroccan authorities in disseminating the use of available electronic tools in public procurement procedures. The electronic portal is now widely used by public agencies to publish offers, terms of reference, and results. In Azerbaijan, two IDFs were approved in 2002 and 2005 for public procurement strengthening and monitoring (for around $205,000) and for strengthening e-government procurement ($101,000). Its IDFs paralleled one DPL—the Second Structural Adjustment Credit—which also included support for procurement reform.  2002: Strengthening of the performance of the State Procurement Agency through an awareness campaign, developing a website, and preparing a training strategy. In addition, support to establish an assessment and benchmarking mechanism to monitor and ensure compliance with the procurement law by procuring entities.  2005: Further support for the implementation of e-procurement system. Training and public awareness components were also foreseen. There were disagreements between the procurement agency and the Bank regarding the selection of a consulting firm to carry out some of the activities, which resulted in delays and cancellation of 66 percent of committed funds. The training and awareness campaign activities were not carried out, and the rest was financed mainly with government funds. Source: IEG. Thus, much of the Bank’s support to procurement capacity building was through its DPOs. In numeric terms, if not in value, these were paralleled by a large number of IDF grants. The percentage of World Bank investment lending specifically addressing procurement was very modest, maybe because of borrower reluctance to borrow for technical assistance, which clients often prefer to receive on a grant basis, as through the IDF. Another factor for the preponderance of DPOs could be the 79 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING emphasis that the Bank’s DPOs put on the improvement of overall public financial management, especially after the introduction of the PRSC instrument (Table 4.1) (IEG 2010e). Table 4.1. Number of Procurement Projects, 2002–11 Any content in % projects High procurement High with high procurement and related procurement procureme content ($ Percent Instrument areas content nt content millions) (of value in $) DPO 402 42 36 3,875 94 IL 193 14 12 237 6 IDF 220 62 53 16 0 Total 815 118 100 4,128 100 Source: IEG. Note: DPO = development policy operation; IDF = Institutional Development Fund grant; IL = investment loan. Finally, much project lending, even if not focused on procurement per se, has been supported by the establishment of project implementation units for the ring-fenced execution of Bank-financed projects, often complemented by capacity-building support for undertaking procurement within Bank operations. Skills thus learned were often transferred to other public and private agencies. Figure 4.1 illustrates the number of procurement related operations, by year of approval, and shows that the numbers depend considerably on the thresholds of procurement content adopted. Nonetheless, by any measure, there was some decline in numbers of procurement-related loans around the years of the financial crisis, and some recovery thereafter, in FY10. Recent lending for procurement may be on the decline again. Both DPOs and investment loans exhibit this pattern, although there seems to have been some secular decline in numbers of IDFs. Figure 4.1. Operations That Focused on Building Procurement Capacity, 2002–11 Source: IEG analysis. 80 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING Note: DPOs with some procurement refers to operations with at least one procurement prior action; there were 202 of these projects during FY02–11. Investment loans with some procurement content are those that have a share of procurement elements in the overall project design that is greater than zero--101 in this period. High procurement content DPOs have more than 15 percent of their prior actions related to procurement (42 high procurement DPOs were identified in the period). Similarly, for investment loans, high procurement projects are those with a share of procurement elements in the overall project design greater than 30 percent; there were 14 of these during FY02–11. Sixty-two IDFs were identified, based on a through text search in the title of the grant. Table 4.2 shows the nature of operations in 103 countries in support of procurement reform. Countries that had projects with some procurement content—365 projects— are shown in the table. In about 60 percent of these countries the Bank used two or all three of the different instruments. Table 4.2. Procurement Portfolio Profile, FY02–11 (number of countries) Sampled Instrument All countries % of total countries % of total DPL 21 20 1 5 IL 5 5 - - IDF 13 13 1 5 DPL+IDF 8 8 3 15 DPL+IL 25 24 9 45 IL+IDF 5 5 - - DPL+IL+IDF 26 25 6 30 Total 103 100 20 100 Source: IEG analysis. Note: DPL = development policy loan; IDF = institution development fund; IL = investment loan. Project data include DPLs, investment loans and IDFs that have some procurement content. DPLs have some procurement content when they have at least one prior action related to procurement. 202 DPLs were identified. Investment loans that have a non-zero share of components related to procurement were included in the table (101 investment loans were identified). IDFs that have some procurement content –62 of them--when procurement is explicitly referred to in the project’s title. The table is therefore based on data for 365 projects that were approved during the period FY02–11 in 103 countries. The sample countries were Albania, Azerbaijan, Bangladesh, Bhutan, Colombia, Ethiopia, Ghana, Indonesia, Lao PDR, Mexico, Morocco, Mozambique, Peru, the Philippines, Rwanda, Senegal, Sierra Leone, Tanzania, Turkey, and Vietnam. Honduras was one of the 21 focus countries but did not have projects with procurement content. Regional projects are excluded from the “all countries” count. CHARACTERISTICS OF BANK LENDING FOR ENHANCED PROCUREMENT CAPACITY BY INSTRUMENT IEG’s desk review of procurement supporting operations illustrated first, some difference in the character of these operations by nature of instrument. Policy-based lending operations aim to support governments for undertaking policy actions that affect public procurement. Not surprisingly, therefore, such institutional and legal actions tend to dominate the nature of support in many DPOs. Box 4.1 and Box 4.4 provide illustrations that made use of the PRSC series to support procurement reform and made use of combinations of different series of DPLs, sometimes including PRSCs, aimed variously at economic rehabilitation, growth, public investment reform, and fiscal and institutional support. Thus support was frequently provided for passing a public procurement code or law, or passing amendments to it (Mozambique PRSCs 1 and 2; Tanzania PRSCs 2 and 8; Sierra 81 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING Leone Economic Rehabilitation series; and Colombia’s Fiscal and Institutional DPOs). Next, there was emphasis on setting up regulatory authorities, preparing implementation plans, supporting the often newly established institutions (often in terms of staffing and training, or enhanced authority), and rolling out reforms at the state or district level (Mozambique PRSCs 2, 3, 4, and 5; Tanzania PRSCs 3 and 4; and Sierra Leone’s second programmatic series for governance reform). Support for implementation of good practice in procurement or for increased transparency was less frequent, although there were some instances (Mozambique’s PRSC 6 supported an increase in the percentage of public contracts that were subject to public tender; Vietnam’s PRSCs supported competitive bidding, a procurement code of ethics, and an open access electronic system). Box 4.4. Supporting Government Procurement Capacity Through Combinations of Programmatic DPLs Sierra Leone had eight programmatic DPOs in two series: the Economic Rehabilitation and Recovery Credits (II–IV) and the Governance Reform and Growth Credit (I–IV). Most had some prior action related to procurement, amounting on average to 2 out of 11 prior actions.  The first series supported the development of a reform program and the enactment of a new legislative and regulatory framework, followed by the establishment of a National Public Procurement Authority, an Independent Procurement Review Panel, and procurement committees in several ministries.  The second series focused initially on strengthening institutions (staffing of the National Public Procurement Authority), adopting new regulations, and requiring ministerial procurement offices to adopt procurement plans for the budget year. The second, third, and fourth operations in this series actually set targets for improvements in procurement plan preparation and quality as well as broader result targets (share of procurement conducted through open competition). Seven of 16 DPOs in Vietnam had a prior action related to procurement, as elements of three different programmatic series: PRSCs (III, VIII, and IX); “Program 135 Phase 2 Support Operation for Communes Facing Extreme Hardship in Ethnic Minority and Mountainous Areas” (II–III); and Public Investment Reform (I–II). On average, 1 in 10 prior actions was procurement related. They focused on:  PRSCs: Open competitive bidding, establishment of a Procurement Bulletin, annual procurement audits, adopting a pilot procurement code of ethics, and launching an open access electronic bidding system. The PRSCs covered many different areas; procurement was one minor element embedded under the broad objective of fighting corruption. Even though procurement actions were 1 in 12 total prior actions, the number of benchmarks was much more numerous.  Program 13: Enhanced scope for community contracting and more transparent and competitive public procurement methods. These measures constituted a part of a project 82 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING for ethnic minorities that had CDD elements.  Public Investment Reform: Issuing a law to clarify that late bids in procurement tenders for public investment projects shall be rejected and issuing guidelines to implement e- procurement in pilot provinces and agencies. The law on procurement was adopted in April 2006 and the Department of Public Procurement was created. The PRSC 9 Project Appraisal Document indicates that progress was made and annual procurement compliance reviews of projects funded by state budget were conducted; the findings of selected audits were made public and a pilot on procurement code of ethics was adopted for participants. Similarly, an open-access electronic bidding system was launched. A 2005 Public Expenditure Review recognized the slower-than-expected progress and listed the remaining challenges, including strengthening the legal framework, establishing a complaint mechanism, and setting up procurement monitoring mechanisms. Three of Colombia’s DPOs, under its Programmatic Fiscal and Institutional series, had procurement elements, with an average of 2 of 11 prior actions related to procurement. They supported the passing of amendments to the procurement law and the creation of a procurement agency in charge of issuing regulations and monitoring the application of the law. After initial delays, Colombia enacted the new law and associated regulatory decrees. However, the creation of the procurement agency was not passed by Congress. Instead, an Intersectoral Commission for Public Contracting was created to act as the agency for establishing common guidelines and for monitoring and evaluating public sector procurement. This body reportedly does not have the full scope of functions proposed in the law and lacks the institutional structure needed for it to be sustainable Source: IEG desk review. Such hands-on attention to capacity development was also evident in other countries that received technical assistance for procurement, even if the loans included other aspects of support to public policy and administration (Box 4.6).73 In Afghanistan, a series of technical support loans led the country from an initial position, where it was essential to hire an external procurement agent for IDA or trust fund purchases, to a situation five years later where local authorities had the competence to deliver the program. The loans also addressed traditional areas such as support to a new procurement support office, as well as more “forefront“ areas such as e-procurement, e-reporting, and e-monitoring. Bank support to procurement in Panama was equally striking. It integrated traditional areas, such as the drafting of a new procurement law, and provided support for a public procurement information system gave strong support to the development of e-procurement under Panama Compra, a document management system, the creation of a supplier registry, and an online research system to inform best practice. 83 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING Box 4.5. Training Programs for Procurement—Training the Trainers in Bangladesh  The Bangladesh program initially sought the development of a critical mass of 25 trainers to help and sustain the in-country training capacity.  The “professional” training stream consists initially of a three-week course on the legislation, and goods, works, services, and consultant contracts. To date, more than 2,600 officers have been through this training.  Short courses (half day) have targeted groups such as policy makers, auditors, Parliament members, and business and bidding community, entry-level, mid-level and senior civil servants have included about 3,000 participants.  Bangladesh training institutions have established link programs with international training institutions to sustain the program.  Retention is an ongoing issue. Retention rates are variable and depend on specific ministries.  Overall proficiency from these programs is not high, as acknowledged by the training entities themselves.  Longer-term financial sustainability may be an issue once Bank support ends, although there are hopes for direct budgetary support in the future. Source: IEG. IDF grants were able to achieve the broader spectrum of objectives that were embedded in larger-scale technical assistance loans, although their small sums (the average size of procurement IDF grants was $250,000) limited the amount that each could achieve. In the Philippines, a series of IDF grants provided sustained capacity building that spanned a range of areas from the setting up of the procurement authority to programs for training, audit, and e-procurement, including training to assist users. Azerbaijan’s IDFs also covered a spectrum of areas from the strengthening of the sate procurement agency to support for e-procurement. However, in this case, some areas were not implemented through the IDF (Box 4.3). Procurement objectives were more clear for IDF operations.74 Box 4.6. Investment Lending/Technical Assistance Projects—Afghanistan and Panama In Afghanistan, an emergency public administration project, approved in FY02 for $10 million, facilitated the creation of a central procurement facility and financed the employment of an international procurement agent for the Afghanistan Reconstruction Trust Fund that would also provide training including on-the-job training and mentoring using a train-the- trainers approach. A follow-up Public Administration Capacity Building Project, approved in FY05 for $27 million, continued direct operational support of the procurement agent and further support for capacity building that targeted staff from 25 ministries, four procuring entities and three provinces. It also supported promulgation and implementation of the new procurement law and regulations, the establishment of a procurement policy unit in government, and the 84 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING finalization of standard bidding documents. Initiation of e-reporting and e-monitoring began. With the $33 million Public Financial Management Reform Project approved in FY07, the Bank continued its support to the newly created regulatory body, drafting more Standard Bidding Documents, a procurement manual, and a guidance note and further developing the website. Further staff training helped prepare for “de-concentration” of procurement in line ministries and provincial finance offices. According to its ICR, the Emergency Public Administration Project encountered teething problems. Many of the ministry staff trained in procurement were not involved by their managers in actual procurements. Later, retention of trained staff became an issue. But by the end of the third operation, the procurement unit was operating on its own and training became self-sustaining under the Afghan Civil Service Institute. A transition from bought-in capacity to government managed systems has been achieved (Public Financial Management Reform Project ICR). In Panama, a Public Policy Reform Technical Assistance Project was approved in FY02 for $11 million, with additional financing of $6 million in 2008. The procurement component amounted to $2 million. The project financed an overall review of existing procurement and related regulations, the strengthening of the procurement directorate, development of guidelines, standard documentation and bidding documents, and a human resources policy and training program for government agencies’ procurement units. It also financed the establishment of a public procurement information system and provided a foundation for the introduction of Panama Compra, including system development and computer hardware and software. It financed the hosting and the security administration of the system; and a call center for users. It also financed the launch of the version 2.0 of the system, which allowed online bidding, in April 2010. The ICR Review points to progress in passing and implementing a new procurement law and implementation of e-procurement, though progress was slower than envisaged. The ICR Review also found partial progress in benchmarking indicators: there was an increase in the awareness of the existence of Panama Compra and an increase in the perception of fairness of public procurement by the general public; improvement of providers’ perceptions of the availability of information and reduction of red tape; other indicators (for example, perceptions of corruption, facilitation, savings) were unchanged. As a follow-up, the Enhanced Public Sector Efficiency Technical Assistance Loan was approved in FY11 for $55 million. This project supported the drafting of a new procurement law to address limitations concerning major infrastructure projects and the fragmented nature of the legal framework. The project also envisaged the development of a business continuity model, a document management system, and the creation of a supplier registry. Finally, the project would support the development of a research center in the Procurement Administrative Tribunal through the provision of hardware and software for the set-up of a virtual library on best practices in public procurement, doctrine and jurisprudence. Per the latest ISR (March 2013), agreements were reached with two universities for the implementation of the Panama Compra Academy, but arrangements are still being defined. The Procurement Administrative Tribunal has identified the model to follow for the implementation of a virtual center. Source: IEG. 85 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING It should be noted that, as mentioned above, in many cases, the Bank’s engagement on procurement reform with any given country straddled a combination of instruments. There are several examples from the portfolio review as well as from the country case studies of useful synergies between these instrument classes (for example, in Afghanistan, Bangladesh, and Panama).75 To some extent, therefore, the differences in character of the instruments highlighted the natural complementarity between policy level actions in DPLs, with actual implementation in technical assistance loans or IDF grants. CHARACTERISTICS OF BANK LENDING FOR ENHANCED PROCUREMENT CAPACITY—IEG SAMPLE IEG undertook an in-depth desk-based analysis of procurement support operations in 20 focus countries. The sample covered around 90 projects, including 68 DPOs, 15 investment loans, and 15 IDF grants.76 Of the 68 DPOs in the sample, 63 were programmatic and 59 were part of a series of three or more DPOs.77 There was relatively broad regional distribution, but some over-representation of Africa, reflecting the large number of programmatic PRSCs in this region. Eight operations were in fragile states, and four in small states. In terms of loan size and amounts, DPOs account for most of the amounts loaned for procurement reform in the sample countries (Table 4.3). Based on this sample, IEG undertook a review of the content of the Bank’s interventions in terms of procurement capacity building through lending. The review used a detailed template with the four broad areas of the MAPS framework, both at the country level and at the project level.78 Results are summarized in Appendix Table D.12. Table 4.3. Procurement Sample ($ millions) Procurement Average share Number of weighted Total projects Average of procurement Instrument type projects amount amount project size content (%) DPO 60 941 10170 169.5 9 IDF 15 4 4 0.3 94 IL 15 58 393 26.2 15 Total 90 1003 10567 117.4 9 Source: IEG Note: The procurement content was not calculated for 8 DPOs that were parts of series; hence the total is 90 and not 98. In terms of the integration of procurement into project design, lending was often underpinned by comprehensive diagnostic work, typically undertaken by the Bank. Procurement activities were reasonably embedded in overall project results frameworks, albeit with weaknesses in assessments of procurement results. However, there was a widespread absence of clear analysis of country capacity to implement procurement aspects.79 86 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING Aggregating Scores at the Country Level Although individual projects often addressed only a subset of procurement issues, combinations of projects in all countries achieved wider coverage. In 14 countries at least, one project addressed the legislative framework; in 18 countries, there were interventions in terms of accompanying regulations and support to institution building (for example, through establishing a procurement agency). However, coverage was less focused on efficiency and market building. In terms of transparency, there were links between the financial management and procurement aspects, through components for internal and external audits, almost invariably combined with procurement components. There was frequent attention to basic training, though less to training of trainers or building long-term training strategies. Attention to fraud and corruption was modest, as was attention to more advanced procurement issues, such as e-procurement. Issues around contract management and administration were touched on in just four countries. In only two of the countries—Peru and Sierra Leone—was there an effort toward the integration of procurement into the budgeting, planning, and auditing processes.80 However, numeric scores alone have limitations and do not indicate the importance that may have been given, in an operation, to procurement-related issues. In Peru, for example, the Bank’s scattered lending support for procurement reform became a key element of the government's stimulus package during the global economic crisis, with a new procurement law, greater institutional clarity, standard bidding documents, and a strengthened e-procurement system.81 Aggregating Scores at the Project Level A review of content at the level of individual projects, based on their stated objectives, yields similar findings. The most important areas for projects have been improvements in the regulatory framework (51 percent of all projects); establishing or strengthening institutional functions (44 percent); strengthening implementation arrangements (43 percent); strengthening internal and external audits or other oversight mechanisms (38 percent); supporting increased transparency and disclosure (34 percent); and strengthening quality of overall legislative framework (33 percent). At times, a legislative framework was already in place prior to the evaluation period, allowing a focus on accompanying regulations and institution building.82 There was much more limited attention to (i) supporting integration into the budgeting, planning, and audit process (6 percent); (ii) supporting the strengthening and dissemination of monitoring and procurement statistics (7 percent); (iii) providing for improved contract administration and management (8 percent); 87 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING improving the complaint process and appeal mechanisms (9 percent); and providing support for identifying and sanctioning corruption and fraud (13 percent). Longer-term capacity development also seems to have been limited. Although 29 percent of projects focused on enhancing capacity and cadres for public procurement and 33 percent provided for in-country training, there was at best modest attention (10 percent) to embedding training activities into sustainable, longer-term training strategies or to provide training for trainers. Focus has thus so far has been on first-round reforms. The Bank’s procurement engagements in these countries have been heavily focused on institutional aspects, with less attention to improving the functioning of procurement, as illustrated by modest percentages for issues like e-procurement (20 percent), strengthening the market for public procurement (18 percent), and improved contract administration and management (8 percent). RESULTS OF EFFORTS TO SUPPORT PROCUREMENT THROUGH LENDING OPERATIONS To assess achievement, results were measured relative to activities supported, based on a results analysis within IEG’s evaluative template.83 A number of projects showed indications of partial progress, in several areas:  Improvement in the strength and stability of the procurement institutional framework (highly achieved in only two cases, modestly improved in 55 projects, and negligibly met in 41 projects)  Paving the way for an increase in the capacity of procurement cadres to plan, manage, and monitor procurement processes (34 cases)  Contributing to improve transparency, integrity, and accountability of procurement and reduce the scope of corrupt practices (43 cases)  Strengthening internal or external audits or other oversight mechanisms (43 projects)  Supporting increased use of e-procurement. Reportedly, some form of expanded e-procurement was an element of a fifth of all reviewed projects and two-fifths of all IDF projects.84 Most of this was focused in IDFs and investment loans with much smaller share for DPLs.85 Therefore, technical assistance with modestly sized, discrete IDF components represented more than half of all interventions for this purpose. Other targeted results were only modestly achieved. For example, for those projects targeting the legislative framework, weighted average result scores in this area were only 1.6 (on a scale of 0–3), with 17 percent showing results to be negligibly met (5 projects) and only 23 percent highly achieved (7 projects)—despite a generally high 88 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING level of attention to this area. In many cases, this demonstrates the slow and incremental nature of procurement reform. Only four projects were able to achieve a high level of improvement in terms of the use of competitive methods; 83 projects achieved a score of 0. Scores for improved transparency, integrity, and accountability also fared poorly in terms of results, achieving a weighted average score of 0.5.86 Probably more could have been done for capacity building. A significant number of projects (33 percent or 30 projects) provided for local training, but only 10 percent embedded training in a sustainable and long-term strategy (9 projects). Bangladesh illustrates a case of good practice in this context (Box 4.5), whereas the Bank in Indonesia had no overall plan or roadmap for training and capacity building. Reported achievements of integration of procurement into the budgeting, planning, and audit processes were also limited (only 13 projects), with only 7 projects providing for improved contract administration and management. Not a single project had evidence of an increased number of timely payments or reduced delays in procurement as a result of better integration into the planning and budgeting process.87 The results clearly show the limited scope for quick change. Modest improvements from one operation to the next (as illustrated above for Afghanistan, Bangladesh, Mozambique, Tanzania, and others) indicate that such policy and institutional reforms take time to be implemented and an even longer time to show results. In many countries, this is the inevitable reflection of limited country capacity in the public sector. The Bank’s strategy of continuous engagement was appropriate but underresourced, given the magnitude of the challenge. And finally, it is clear that government commitment is paramount for success.88 This is also illustrated in the case of the Mozambique PRSCs, where by and large the individual projects gave modest attention to procurement, and more focus on institutional steps. In spite of particular cases, the evidence is not conclusive regarding the synergies between DPOs and IDF grants. Examples of such a synergy seem to be present in Azerbaijan—potentially because of timing of both instruments—and Turkey, and especially in Bhutan, where such grants and also technical assistance lending have supported DPOs with strong emphasis on procurement. There is limited experience with specific investment loans for procurement. In Bangladesh there were two dedicated specific investment loans: the first covered a broad range of issues; the second (ongoing) was more of a training program. The procurement issues addressed in the two loans have also in parallel been covered in four DPLs, with some modest institutional components but otherwise not addressing procurement. 89 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING EVIDENCE FROM COUNTRY CASE STUDIES Many conclusions from the desk review are reinforced by evidence from country cases studies, which also shed light on additional aspects of capacity building by lending, notably for loans not targeted at procurement reform. Targeted Lending for Procurement Reform—Mutually Reinforcing Loans Interlocking and mutually reinforcing operations were perceived to be particularly beneficial by country clients. Field staff in Bangladesh pointed out that there were two DPLs that had procurement components (one with a condition on the drafting of the procurement law and the other on the enactment of the law), and in parallel, four PRSCs offered indirect and “interlocking” support to procurement reform via triggers relating to anticorruption measures. These operations helped build a basis for the success of the two procurement-dedicated investment loans. DPLs and PRSCs were especially important because of their higher visibility, including discussion in budget sessions of Parliament. In Tanzania, there were strong elements of procurement reform in most PRSCs, but there are no procurement triggers in PRSC 9. Yet discussions suggest that these could be incorporated in a future DPO on governance and accountability, and there is also a DPO on reforms in certain sectors that focuses on procurement issues. In Peru, following the crisis-period DPO on fiscal support that focused considerably on procurement, country respondents mention an IDF grant under preparation for building procurement capacity, intended for 2013. In Morocco, strengthening of the procurement system, although not expressly included in the Bank country strategy, has been constantly discussed with Moroccan authorities, especially on the basis of the analysis conducted within the framework of the use of country systems study.89 Several IDF grants supported procurement reforms and strengthened capacity at the national procurement agency. Support to Morocco in developing e-procurement was also included in the fourth Public Administration Reform Loan, and related policy actions are envisaged in a forthcoming DPL on accountability and transparency. Targeted Lending for Procurement Reform—Slow Pace and Mixed Results In Azerbaijan, the Bank supported a $60 million structural adjustment credit in 2002 aimed at a range of public sector reforms, including transparency and accountability in public financial management. Prior actions for procurement included the passage and implementation of a public procurement law and other actions described in the CPAR action plan. The need for implementation support led to the subsequent 90 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING approval of an $8 million technical assistance project in 2009 that included components for procurement, as well as a series of IDF grants (1999–00, 2003 and 2005). The third IDF grant for $299,000 aimed to develop capacity in e-procurement but did not fare well because of government reluctance to employ international consultants. In Indonesia, eight DPLs included procurement reform triggers, though at the insistence of the government, the most recent did not. Establishment of the procurement regulatory body was a DPL trigger, as was the establishment of a legal framework for public procurement. Although it did not achieve the goal of a procurement law, the latter helped improve the decrees that govern procurement. DPLs 7 and 8 and also included elements of capacity building to develop a cadre of procurement professionals. The government and Bank staff concur that DPL triggers helped move the procurement reform agenda forward. Procurement triggers changed over the years, from addressing policy issues to more practical concerns. The latest DPL sets operational targets for the use of e- procurement functions. This is a technical matter on how to operate, in contrast to the 2005 requirement to set up the public procurement authority—a shared goal of all development partners and the government. Earlier DPLs were perceived to be more effective as the reform agenda was starting, whereas today they may be a less suitable vehicle for the implementation phase. The government informed IEG that in view of its progress, it no longer “needs to be told what to do.” Indonesia had no targeted technical assistance loan; however, under investment lending, it is standard practice for the Bank’s procurement unit to provide training to all project implementing agencies. Targeted Lending for Procurement Reform—Missed Opportunities? In Turkey, although there were two DPLs in the areas of public finance and economic management, these did not include triggers for public procurement reform, perhaps because of Turkey’s preference to work with the European Union in this area. Further, Turkey did not have any investment lending that was especially devoted to procurement reform. The Bank continued to support capacity building, though this occurred because of interactions between procurement staff and government officials charged with the implementation of Bank investment lending projects. The extent to which knowledge was imparted in this process was deemed significant, but narrowly limited to project implementation alone. Although Turkey’s procurement system 91 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING made good progress in the early years of the last decade, both suppliers and government point to many areas where further improvement would be desirable. Support to Project Implementing Units During Project Implementation Overall, country clients perceived the value of Bank support to capacity building, through any form of lending, to be quite substantial (with a score of 3.4 out of 4, on average)—although Bank procurement staff deemed its contributions to be more modest (with a score of 2.3).90 Almost across the board, country case study evidence suggests that a very important vehicle for supporting procurement capacity development is the day-to-day interaction between clients and Bank procurement staff on procurement issues during project implementation. At the most basic level, the within-project training and hands-on implementation support provided by World Bank procurement staff was not adequately resourced—in only one country visited could a separate earmarked work program agreement allocation be identified for procurement capacity building, separate from transactional procurement work of the procurement unit (Box 4.7). Box 4.7. Building Capacity Through Nontargeted Investment Lending In Ethiopia, procurement capacity building related to project implementation and strengthening of implementing agencies’ capacity is provided in all investment loans, according to the needs of the executing agency. Country management perceived a great need for “…regular training on complex procurement and local-level procurement.” Unfortunately, management also perceived that the “…procurement unit does not have enough staff and resources to do regular training.” Despite resource shortages, specific training courses were initiated for project implementing unit staff at the federal level. In Azerbaijan, procurement capacity building takes place within most Bank investment lending projects. Bank procurement staff train project implementing unit staff as a routine practice in the course of projects, and funds are routinely earmarked for procurement training of implementing agency staff within each investment loan. These components of nontargeted investment loans were reported to be an important instrument for building human resource capacity in procurement. It was reported that “At the moment, the training on-the-job on Bank procurement received by project implementing unit staff, in the course of the investment loan project, appears to be by far the most effective form of capacity building available.” In Indonesia, although training was imparted in a similar way, its long-term value was more debatable. It was commented that there are incentives for government staff to be assigned to project implementing units (allowances, vehicles, and so forth). There are difficulties with sustainability. Knowledge is not readily transferred. Trained staff do not routinely transfer to other Bank-funded projects in the same implementing agency (they go to other assignments or other projects), and the knowledge of Bank project work is lost. Source: IEG country case study questionnaires. 92 CHAPTER 4 SUPPORTING PROCUREMENT CAPACITY THROUGH BANK LENDING By its nature, within-project support is fragmented, and although it was greatly appreciated, the implementing agencies conveyed that it was no substitute for systemic support for national-level systems and institutions. They also conveyed that the Bank’s procurement staff was very stretched and, despite good intentions, could not provide the hands-on support desired because of the inevitable precedence of transactional work. As a result, in some cases, the long-term capacity building achieved for the country was questioned. 93 5. Integrating the Bank’s Procurement Support Governments are under pressure to improve their public financial management and procurement systems to make more effective, efficient, and accountable use of public resources.91 Within the Bank, this requires recognition that such improvement requires work throughout the expenditure management chain, moving away from a focus on integrity in the supplier selection in public procurement and toward a focus on effective management of public resources. Under such a framework, procurement decisions are not relegated to the end of the expenditure chain but brought forward to budget preparation, with greater attention to procurement planning. Building awareness of such needs and helping implement them in practice requires coordination in Bank work practices, especially across the PREM and Operations Policy and Country Services (OPCS) financial management and procurement groups. These groups provide most of the Bank’s capacity building support, as discussed in Chapters 3 and 4. The World Bank Institute (WBI) complements this with smaller levels of support for procurement capacity development. Previous IEG work has pointed to the need for better internal coordination in terms of diagnostic work among the various units responsible for public financial management, including the PREM, financial management, and procurement groups. IEG has also pointed to the need to prepare integrated action plans for clients (IEG 2008b). And as discussed in the preceding chapter, there is a parallel need for integration in DPOs. In regard to investment lending, operations focused on procurement have been scarce, so it is difficult to comment on collaboration. In this chapter, IEG first reviews coordination between the OPCS procurement anchor and PREM staff; it then examines the role and contributions of the WBI, to see whether these complemented or overlapped with work under way in the procurement anchor. Main Findings  There are anecdotal examples of good collaboration; however, there are also instances where collaboration has been limited.  Good practice usually reflects one or more similar elements: ample funding, support from governments and senior regional management, and staff from different practices that work well together.  It is desirable to institutionalize collaboration through better incentives and rewards for knowledge sharing and cooperation. 94 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT  Preparation and dissemination of information by the Bank on how good procurement actually helps expenditure management would help motivate staff as well as clients.  WBI made valuable contributions to developing procurement capacity, and there was good coordination with the Bank’s procurement anchor.  Many services were demand driven, and WBI exercised selectivity in choosing its entry points and instruments.  So far, WBI’s open contracting work is on track to achieve potential impact. This work has been on a limited scale, because of resource constraints, but it could be scaled up through partnerships supporting open contracting data standards.  There is scope for expanding the effective use of PREM and WBI-led public expenditure management tools to incorporate information relevant to procurement.  WBI’s e-procurement training courses have included relevant material, although WBI could strengthen its modes of delivery. Country clients expect Bank procurement staff to provide leadership in areas such as e- procurement.  Some aspects of the initiatives of PREM and WBI are highly reliant on trust fund support, which makes sustainability questionable.  Despite careful management of work to avoid obvious overlap, it remains a question whether the present distribution of work across PREM, WBI, and the OPCS procurement anchor facilitates maximum effectiveness from the perspective of developing country procurement capacity. Collaboration: Efforts and Outcomes PREM, OPCS, and WBI Working in an Integrated Manner Anecdotally, there are many examples of commitments to improve coordination among the three practices.92 Moving from statements of intent to actual practice, IEG found 12 examples of good practice cited by one or more of the 43 staff interviewed, covering operations approved over the period 2001–12, with two scheduled for approval in 2013. They included three investment loans, two development policy loans (including one at the sub-national level), two program for results projects, four AAA outputs, and one internal Bank training exercise. These examples of good collaboration across practices tend to have one or more similar elements: ample funding, support from governments and senior regional management, and staff from different practices that personally enjoy working with each other. 95 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT One example is the ongoing PREM-led project on procurement in Indonesia, which brings together staff from the two practices and the WBI. PREM has taken the lead on this through its access to a multidonor trust fund on public financial management for Indonesia. OPCS staff have participated in joint missions, with good coordination in resolving problems and in applying their expert knowledge of transactions and financial systems. PREM staff have helped with statistical and other analytical tools to understand the underlying political economy causes of patterns observed. The Ministry of Finance encourages this collaboration, seeing the need for multidisciplinary skills to analyze the links between budget execution and service delivery, and how in poor districts this leads to declining procurement spending. Examples of collaboration in lending operations are provided in Appendix Table E.1. Examples Where Coordination Could Be More Effective There are also examples where the practices could coordinate more effectively. Attempts at integrated diagnostics have been relatively rare, for example, the move from separate CPARS and Country Financial Accountability Assessments to Country Integrated Fiduciary Assessments, as well as the move toward joint Public Expenditure Management and Financial Accountability Reviews. As yet, integration is not complete. For example, the Country Integrated Fiduciary Assessment mainly presents public financial management and procurement sections and actions separately. There is also a need for more granular studies on specific topics such as financial management information systems, budget planning, and audits that try to determine how to best undertake integration in these specific areas. Collaboration has been only partially effective with regard to the PEFA instrument. The PREM Public Sector unit houses the Secretariat, which splits the leadership of teams with the procurement anchor. Both practices collaborated in developing the PEFA instrument and rolling it out, especially indicator PI-19. However, when it was first designed in 2005, the Bank’s procurement anchor reportedly questioned the value of reducing the complex processes of procurement to a single, high-level, three-dimensional indicator. Discussions finally led to a reformulated PI-19 with four dimensions.93 Although it was considered an improvement, some procurement staff still complain of superficiality, and the PEFA has been little used as a key diagnostic instrument in this area. Finally, it should be remembered that practices, in terms of collaboration, vary considerably across the Bank’s regions. 96 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT Possible Ways Forward Calls for improved collaboration between PREM and OPCS are not new (World Bank 2005a; IEG 2011d). Interviewees offered a range of suggestions for further strengthened collaboration, including better incentives and rewards for knowledge sharing and cooperation and consistent messages from managers that both recognize technical specialization and encourage cross-network collaboration. As IEG put it in a recent evaluation: Management should revamp the system of formal incentives for individuals to ensure that middle and senior management and staff are assessed on performance, including (a) on sharing of knowledge and collaboration across sectors, and (b) the quality and results of operations for which they are responsible; and reward performance that goes beyond Board approval for lending operations, while being more transparent and better linking career advancement decisions to quality and achievement of development objectives (IEG 2012a). Preparation and dissemination of information by the Bank on how good procurement actually helps expenditure management would help motivate staff as well as clients (as discussed in Chapter 4). 94 There is some information on public financial management reform leading to better Country Policy and Institutional Assessment scores. One study finds that two-thirds of governments that benefitted from public financial management support from the Bank improved their CPIA score on budget and financial management between 1999 and 2006 (IEG 2008b). Finally, skill gaps need to be addressed across the three practices. And all interviewees stressed the need to engage in one public financial management and procurement dialogue with senior level borrowers. SUPPORTING COUNTRY PROCUREMENT CAPACITY THROUGH THE WORLD BANK INSTITUTE Bank contributions to the raising of global awareness of the importance of procurement, as well as contributions in some important new areas of procurement practices, have been largely channeled through the WBI. This has been done through training, building communities of practice, and developing tools for the analysis and greater transparency of information on public procurement. WBI complements the Bank’s procurement anchor mainly through its collaborative governance practice, which uses its trust funds to promote open public procurement, including third- party procurement and contract monitoring. IEG evaluates WBI’s contribution to procurement reform in terms of relevance of design, in terms of instrument choice and relevance of outcomes, in terms of country 97 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT conditions and Bank diagnostics; and effectiveness in terms of realizing desired outcomes, to the extent that there has been sufficient time for the objective to be achieved. World Bank Institute Instrument Choice WBI’s four principal forms of support directly related to procurement include contract monitoring, political economy analysis, e-procurement training, and support for public expenditure management with potential links to procurement. 95 The choice of these areas suggests reasonable selectivity. Bringing together stakeholders including civil society and the private sector for contract monitoring is good practice in procurement. it contributes to the guiding principle of transparency and eventually enables countries to undertake better oversight of public procurement themselves.96 Also, from the public finance perspective, PEFA indicator PI-19(iii) measures the extent to which key procurement information (government procurement plans, bidding opportunities, contract awards, and data on resolution of procurement complaints) is made available to the public. MAPS indicators 5a and 11a measure similar things. The second area, political economy analysis (in supporting procurement reform in project lending) is a key feature of the Bank’s governance and anticorruption strategy (IEG 2011d). Third, e-learning falls squarely in WBI’s mandate, and its use for supporting e-procurement is critical for supporting efficient market development.97 Finally, WBI’s support to public expenditure management with potential links to procurement is in line with other efforts across the Bank to foster such linkages. Given resource constraints, these choices reflect strategic focus to maximize impact. Outputs and Outcomes Achieved—Contract Monitoring WBI’s contributions to bringing together civil society, the private sector, and governments in contract monitoring include work on nine African countries (Ghana, Kenya, Liberia, Nigeria, Rwanda, Sierra Leone, Uganda, Tanzania, and Zambia). In Asia, through Affiliated Networks for Social Accountability, WBI is also working with the Philippines, Mongolia, Indonesia, and Cambodia.98 The selection of countries is based on country office demand and is led by procurement specialists who monitor the effectiveness of the interventions, in terms of quantitative measures, such as the number of contracts monitored, numbers of advocacy actions taken, as well as more abstract measures, such as the extent to which coalitions are strengthened. Evidence of this progress includes new members, fundraising, or active participation from government and applying organizational network analysis of African coalitions (Box 5.1). 98 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT Box 5.1. Developing Instruments and Markets to Promote Open Contracting WBI carried out a self-assessment of outcomes of its trust-fund financed work in February 2013, including documented outcomes in three of nine participating African countries, two of four Asian countries participating through Affiliated Network for Social Accountability, and progress reported with a global partnership: the Open Contracting Lab (Appendix Table E.3). Results, which are being disseminated through both publications and through a community of practice, suggest that the open contracting work has achieved the desired impact in some countries and is on track to achieve potential impact in others. Evidence also suggests cooperation among civil society, the private sector, and government. A known limitation of such work is that civil society groups generally lack the technical knowledge to understand the intricacies of the procurement process. There are rare cases— for example, a civil society group in Senegal with a trained board of directors—of such groups having the technical knowledge to affect detailed processes.a Nevertheless, building awareness of potential for abuse can help protect against fraudulent or corrupt practices. Although this work has only been carried out on a pilot scale in a small number of countries, there are potential opportunities for scaling up through the promotion of open contracting data standards, promoted through crosscutting platforms like the G8, the Open Government Partnership, and the OECD Effective Institutions Platform. Source: IEG interviews. a. Dissemination through the community of practice is available at http://pro-act.org/. WBI also supported an international competition that led to the preparation of 60 case studies on innovative procurement reforms, publishing results of the best entries (see http://pro-act.org/). Outputs and Outcomes Achieved—Political Economy Analysis WBI carries out political economy analysis in support of all streams of its procurement work. The largest piece of work to date is an ongoing study on procurement at the local government level in Indonesia, done in partnership with WBI, PREM (Bangkok governance hub), the Bank’s procurement anchor, and the Asia Foundation. The study is highly relevant to the Bank because large CDD projects have posed special procurement challenges to local governments. The objective is to examine how to improve the volume and quality of procurement-related spending at the provincial and local government level. The study looks at the extent to which local level spending is constrained by slow and poor-quality procurement and provides recommendations on how those constraints could be reduced in the short to medium term. It identifies local governments that have achieved some traction with procurement reform and elaborates the underlying drivers that helped this reform. The study team has pulled together data on the institutional set-up, business processes, capacity building, and civil society monitoring as of 2011. The study 99 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT design is built on the assumption that the main impediment to procurement reform is not a lack of capacity in subnational governments, but the local political and economic incentives against reform.99 Outputs and Outcomes Achieved—Training on E-Procurement Since 2009, WBI has supported a series of e-procurement training courses, each building on the previous one and including an increasing amount of virtual material in successive series to allow greater scale with available resources. Initially, training provided examples of the benefits of e-procurement and case studies of previous successful transitions to e-procurement, through an online knowledge bank. Later modules provided more detailed information on available information technology models used around the world and business models for managing the development, implementation, monitoring, and regulation of e-procurement systems. Finally, the training sought to offer hands-on help drafting strategic action plans and building global communities of practice. The platform is being developed in coordination with other MDBs, so that a common approach will be used. Country teams have reported to WBI that there has been impact at the country level from the training, for example, in the adoption of a customized e-procurement system by Vietnam. Good ownership is evidenced by high participation rates throughout the courses and more nominations for courses than there is space available, despite the lack of typical incentives like overseas travel and per diems. IEG reviewed a hard copy of the course materials and found it to be of high technical quality. The costs are reasonable and are continually being reduced on a per participant basis. The partnership with other MDBs in developing and rolling out the e-learning modules is another positive sign. Although WBI is still in the process of refining its pedagogical techniques, e-learning has the advantage of being scalable. Outputs and Outcomes Achieved—Public Expenditure Management and Other Areas Certain tools of public expenditure management developed jointly in the Bank by the PREM anchor, the Europe and Central Asia region, and WBI for public expenditure analysis could also be extended to assist with collecting information that is useful for procurement benchmarking. Notable among these is the BOOST data tool, which gathers expenditure data directly from a country’s treasury system at the most disaggregated level available and has high potential as a tool for comparative benchmarking of specific expenditures. Some countries, such as Kenya and Moldova, also make BOOST data available to citizens’ groups, helping them to sharpen their advocacy for improved public services. 100 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT The contribution of BOOST for work linked to procurement has been scarce to date, as its original goal was to support Bank teams and governments in completing PERs, which may not include significant new work on procurement. Until recently, BOOST worked with limited funding, although it now has support from the Gates Foundation. It still remains a new tool, yet to develop full potential, with issues of data reliability and the limited capacity of government officials and civil society to use BOOST for data analysis (see Box 5.2). Box 5.2. Performance Reform in Moldova—Using BOOST to Monitor Public Expenditures Procurement issues have a high strategic relevance to the World Bank in Moldova. After a recent major misprocurement, the Country Partnership Strategy included increased competition in public procurement as one of its outcomes, measured as the increase in percent of value of all contracts awarded through competitive processes. Key milestones are that all required secondary procurement legislation be in place, the public procurement department strengthened, an electronic procurement system implemented, and appropriate amendments introduced into the primary legislation. Moldova provides an example of how BOOST has been used to support procurement- related reforms. A district-level analysis of education expenditures in 2009 compared spending and spending efficiency (based on costs per pupil) with need (based on a national deprivation index). The results were mapped to distinguish high-performing from low- performing districts and to establish factors leading to high per student spending. This granular analysis led to reforms supported by the Bank, including better use of expenditures such as fuel for heating (a major procurement item). There has not yet been any impact directly related to procurement, although the tool could be used to assess this. BOOST reportedly has high ownership by government and civil society, with a dedicated unit in government managing the system, and updating real time data on the Web: http://www.mf.gov.md/ro/BOOST/. Sources: IEG interviews; World Bank 2011b, 2012j. Finally, there is other work supported by WBI that is linked or that could be linked to procurement support in the future, but that has not been assessed in detail here. For example, WBI's electronic network of pharmaceutical professionals (http://www.enepp.net/) works to strengthen transparency and accountability on medical procurement. WBI’s support to the Extractive Industries Transparency Initiative also has potential links to procurement, and WBI’s support to building capacity for PPP transactions is an important parallel to Bank work in this area. In addition, WBI is in discussions with the International Budget Project on developing an Open Procurement Index that could be evidenced and promoted along with the Open Budget Index. 101 CHAPTER 5 INTEGRATING THE BANK’S PROCUREMENT SUPPORT OVERALL CONCLUSIONS IEG’s review of select facets of WBI contributions to procurement illustrates significant areas of support directly related to procurement. Based on its analysis, issues of relevance have been addressed, there is limited overlap with the Bank’s procurement anchor, and many services were demand driven. Thus, WBI did exercise reasonable selectivity in choosing its entry points and instruments. WBI’s open contracting work has had the desired impact in some countries and is on track to achieve potential impact in others. However, work has been on a limited scale, because of resource constraints under the present pilot phase. Despite promising achievements, it is not clear how this work can be scaled up to impact more countries. The largest piece of political economy work to date is an ongoing study on local procurement in Indonesia, which is highly relevant for the Bank's current CDD program, as such projects have posed special procurement challenges to local governments. The issues are also relevant in the context of the government’s decentralization reforms. WBI’s e-procurement training courses have included relevant, appropriately phased material, each course building on previous ones and scaling up successfully within available resources. Country teams have reported to WBI that the training has had impact at the country level. However, WBI’s approach could learn further from the most innovative practices of the market. WBI has supported the BOOST data tool, in partnership with PREM and an increasing number of country teams, primarily for public expenditure management, although it has potential to develop links to procurement, albeit likely issues of reliability of disaggregated data and human capacity. Despite careful management to avoid obvious overlap, it nevertheless remains a question whether the present distribution of work across PREM, WBI, and the OPCS anchor facilitates the achievement of maximum effectiveness from the perspective of developing country procurement capacity. Country clients expect Bank procurement staff to provide leadership in areas such as e-procurement. There is scope to expand the effective use of PREM and WBI-led public expenditure management tools to incorporate information relevant to procurement, for example. Some aspects of the initiatives of PREM and WBI rely strongly on trust fund support, which makes sustainability a question. A review of the scope for and areas of closer integration and cooperation could suggest channels for expanded synergy. 102 6. Building Global Good Practice in Procurement Apart from building capacity in its client countries, to what extent has the Bank been effective at building global capacity and good practice in procurement? The need to strengthen and mainstream procurement functions has been emphasized in successive global fora on development effectiveness—the Declarations of Paris, Accra, and Busan. One aspect of such global capacity building is the strengthening and harmonizing of procurement practices among multilateral and bilateral organizations, both at a policy level, in terms of framing common approaches to procurement, and at a project level, in lending. IEG’s assessment of the Bank’s role in this regard includes semistructured interviews with Bank procurement staff knowledgeable of the Bank’s harmonization work and with current and former external participants in various harmonization forums. It also includes documentation produced by these forums and inputs from stakeholders through IEG’s 11-country survey. As emphasized by the high-level fora, a core element of increased development effectiveness of donor country or agency lending has been the greater use by development partners of country systems, including procurement systems. Indicators for tracking the use of country systems were developed as part of the monitoring of the Paris Declaration, and the Bank launched an ambitious exploration of the extent to which country procurement systems could be used for World Bank operations. IEG evaluates the approach adopted by the World Bank to support the use of country systems in procurement through (i) a detailed analysis of the analytical work undertaken in the 20 “pilot” countries for the use of country systems; (ii) a comparison of the World Bank’s approach with that of other development partners; and (iii) a review of stakeholder views in client countries regarding the value of adopting country systems. Main Findings  Over the past 15 years, the Bank has played a lead role in advancing the global agenda of building and harmonizing good practice in public procurement, with many successes. The Bank’s work is widely recognized and appreciated by development partners and client countries.  The Bank has been particularly instrumental in developing assessment tools, starting with its own CPAR, as well as helping to develop the MAPS tool. 103 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT  Efforts to mainstream procurement reform within the context of financial management reform have been less successful.  The Bank’s efforts to coordinate with global partners in the field, on reform, capacity building, and alignment of practices is varied, substantial in Mexico and the Philippines (where the Bank leads a multi-donor coordination effort) but negligible in Indonesia and Turkey (where the Bank has relinquished the lead role to AusAID and the Millennium Challenge Corporation).  Practices of the Bank and other major donors are largely aligned, with differences mainly related to the Bank’s insistence on worldwide procurement (some regional development banks and bilaterals restrict the source of supply) and sanctioning procedures. Although there are few differences, they can be problematic on cofinanced projects where the Bank requires the use of its procedures.  Additionally, the Bank has also been criticized by some multilateral partners for a perceived unwillingness to engage in dialogue with the private sector. Contributing to Global Good Practice—OECD-DAC and the Aid Agenda Two forums have facilitated the Bank’s procurement harmonization efforts over the past 15 years. The first falls under the auspices of OECD-DAC and is broadly represented by a number of multilateral and bilateral organizations, as well as developing country partners. The second is the MDBs’ Heads of Procurement Working Group. Additionally, the Bank has worked with client countries to harmonize procurement practices across government agencies as well as to align country practices with those of the Bank and other donors. IEG’s review focuses on these aspects of the Bank’s harmonization work. THE PROCUREMENT ROUNDTABLE INITIATIVE—2003–05 In 2003 OECD-DAC and the Bank set up the Procurement Roundtable, a first collaborative effort between the Bank and OECD in procurement. The Roundtable subsequently grew to involve 52 bilateral donors, MDBs, United Nations organizations, and representatives from the client countries served by these organizations. The stated objective was "strengthening procurement capacities in developing countries." The Roundtable prepared a document for consideration by the Paris High Level Forum as to how procurement might contribute to development effectiveness, working with countries and bilateral and multilateral donors to develop a set of tools and good practices to improve procurement systems and outcomes.100 104 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT Box 6.1. Strengthening Procurement Through the Roundtable Working Groups The Procurement Roundtable addressed three broad areas:  Mainstreaming. Recognizing that the importance of good procurement was not understood within governments and that high level support for effective reforms was weak, a working group was set up to promote recognition of procurement as a core function of government and a strategic activity, integrated with other public financial management institutions and processes. The working group produced a strategic framework for mainstreaming and strengthening public procurement (OECD-DAC and World Bank 2004). This document emphasized the need for a communications strategy on the importance of procurement reforms, illustrated the important linkages between procurement and public financial management systems and showed how reforms to both need to be coordinated and sequenced.  Capacity Development. From its inception, the Roundtable set out to avoid programs that were low level, short term, donor driven, and/or which failed to address barriers to change. A second working group produced good practices for capacity development (OECD 2006a). The document stressed the importance of accurate assessments, the need for a common understanding of best practice, and the need for appropriate evaluation and monitoring.  Benchmarking, Monitoring, and Evaluation. The Roundtable understood that there was no common agreement among developing countries and among donors over what constituted procurement best practice, and how this should be measured. The lack of consensus hindered progress on remedial reforms. The working group offered three tools to help the process: (i) a baseline indicator tool to assess the strengths and weaknesses of national procurement systems, (ii) a complementary set of monitoring and reporting tools to measure and manage ongoing performance at the national level, and (iii) an assessment tool to evaluate performance at the procurement entity level. These three tools permit governments to assess the impact of any reform initiatives and detect problems of reform design or implementation so that remedial action can be taken. Source: OECD 2005b. The Roundtable was able to establish a consensus among a wide group of stakeholders as to what constituted “best practice” (Box 6.1). This was a significant accomplishment. Unlike financial management, which relies on well-established accounting standards, no such standards existed for public procurement. The Roundtable was also a first effort at international coordination. The Bank played a lead role. IEG was told that the Roundtable most likely would not have happened without the Bank’s leadership and resources. 105 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT OECD-DAC COLLABORATION FOLLOWING THE PARIS DECLARATION OF 2005 Following the Paris High-Level Forum in January 2006 the OECD-DAC Working Party on Aid Effectiveness established the Procurement Joint Venture to carry on the work of the Roundtable.101 These platforms for the incorporation of procurement as a central element of development were sustained through subsequent high-level fora, including the Busan summit of 2011.102 Though the fate of the Procurement Task Force following the Busan summit is unclear, some member countries and institutions of the task force attempted to form a follow-on initiative, the Global Network on Government Procurement.103 The Paris Declaration put a strong emphasis on use of country procurement systems, in line with a general focus toward country systems in many other respects (including financial management). It committed donors to use country systems as the first option for aid programs and if they did not, required them to transparently state the rationale. The World Bank embarked on an ambitious pilot to expand its use of country systems in procurement, in line with the Paris Declaration. The monitoring indicators of the Paris Declaration included specific indicators on procurement in aid practices (Box 6.2). Box 6.2. Paris Declaration Monitoring Indicators on Procurement  Paris Declaration Indicator 2 covers reliable country systems and measures the “number of partner countries that have procurement and public financial management systems that either adhere to broadly accepted good practices or have a reform program in place to achieve these.”  Paris Declaration Indicator 5 covers use of country procurement systems and measures the “percent of donors and of aid flows that use partner country procurement systems which either (a) adhere to broadly accepted good practices or (b) have a reform program in place to achieve these.” Source: OECD 2011c. During the last decade, the procurement working groups that worked in tandem with the high-level fora made significant contributions to supporting global good practice in procurement.104 Although the Bank’s collaboration with the OECD on procurement reform appears to be dormant, after eight years there is some possibility that it will resume once post-Busan structures are in place.105 There were two significant accomplishments of this eight-year collaboration:  Benchmarking, monitoring, and evaluation. This was one of the three objectives initially identified by the Roundtable. The Roundtable initially developed the Baseline Indicators methodology, subsequently refined by the 106 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT Joint Venture and Task Force as the Methodology for Assessment of National Procurement Systems (MAPS—see Chapter 2). MAPS is an important accomplishment for several reasons. First, it established yardsticks for “best practice.” Second, it supported capacity building, acting as a baseline benchmark and providing tools to measure progress. Last, it became integral to pursuing the objectives of the use of country systems agreed to by the Paris Accord. MAPS is the most accepted methodology to assess the overall state of a national procurement systems.106  Capacity development. This was another objective of the Roundtable, which carried through to the Joint Venture. The United Nations Development Programme, in partnership with Danida, launched the UN Procurement Capacity Development Centre in January 2008. The Centre credits the idea of its existence to the OECD-DAC/World Bank Roundtable.107 It sustains the capacity development goal of the Bank’s collaboration with OECD-DAC and actively coordinates and conducts procurement capacity development work. The efforts of the OECD and the Bank to harmonize the field of public procurement reform and to set standards for global good practice in the area of procurement are seen positively and as generally successful. The broad level of participation in the various fora, the wide acceptance of the MAPS tool, and the sustained institutionalization of capacity development attest to this. Many countries also credit the capacity building work as important to reform efforts. But three areas have been singled out for criticism:  The goal of mainstreaming was only partially achieved. Much still needs to be done to integrate procurement reform within the public financial management framework. Feedback on this is mixed. One view is that the task force did attempt greater integration, especially following the Paris meeting, and after Busan when the Working Party on Aid Effectiveness brought the two under one roof, along with results and monitoring. But it was difficult to find common ground, from both sides—there was not much of substance to coordinate on. Others thought that the task force was not sincere in its efforts and that the Bank in particular could have made a stronger effort at closer integration.  There was insufficient engagement with the private sector. Discussions were conducted via the Business and Industry Advisory Committee of the OECD, the independent international business association that advises OECD, but these were few and did not lead to agreements. Companies from developed countries, primarily the United States and Europe, have strongly opposed 107 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT the use of country systems in procurement and lobbied hard against it. The companies thought it would water down procurement procedures as mandated by the Bank’s Guidelines and thereby reduce their business opportunities or render them less secure. Given this strong opposition, it is hard to see how the task force could have done better. However, the view remains that OECD was keen to work with the private sector but that the Bank resisted.  Although MAPS is recognized as a very good tool to assess the structure of a country’s procurement system, it is not considered adequate at assessing how well the structure functions in practice. This was recognized from the outset. It was intended that along with development of the baseline indicators, a set of compliance and performance indicators (CPI) would also be developed. The CPI were never fully developed and are generally recognized as needing further improvements. Some countries (Chile, Ethiopia, Indonesia, and the Philippines) have worked independently to improve the CPI. This risks possible divergence from the goal of developing a standardized and commonly used tool, but is seen as very much within the spirit of the joint venture for countries to develop tools to meet specific needs. The Heads of Procurement Group The Heads of Procurement group started small and informally in 1997 when two procurement specialists from the World Bank and IDB started working on a standardized and harmonized set of documents for the bidding of goods contracts. Differences in bidding documents developed by executing agencies, often for the same project as well as across projects, were found to be time consuming, and the variations across agencies and projects confused bidders. The collaborators believed that standard bidding documents would make procurement more efficient and that a document harmonized across the two Banks would introduce consistency, reduce ambiguities, and generally contribute to the goals of economy, efficiency, transparency, and fairness. Other MDBs joined the process, initially including the African Development Bank, ADB, and European Bank for Reconstruction and Development and later expanding to the Black Sea Trade and Development Bank, European Investment Bank, and Islamic Development Bank. As the group membership grew, its work expanded into other areas of procurement that could benefit from a joint and harmonized approach—electronic procurement, procurement involving the private sector, environmentally responsible procurement, and sanctioning are examples. 108 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT Unlike the OECD-DAC collaborations, the heads of procurement group restricted participation to MDBs or regional organizations (the Nordic Development Fund participated until it was wound down, for example). Bilateral organizations and client countries do not participate. The group is consultative—members are not bound by its policy recommendations. An MDB member is free to choose if, or to what extent, it wishes to adopt outputs of heads of procurement working groups into its procurement policies and practices. This group works informally, setting up working groups as needed. It does not maintain a website and does not publish externally. It meets about every nine months, with working groups meeting as necessary. The standard bidding document working group within the heads of procurement group continues efforts to harmonize and standardize procurement documentation, especially model bidding documents. Heads of procurement group members then adopt the model documents, adjust within agreed parameters, and publish them as their own standard bidding documents. The Bank Guidelines—and those of most MDBs—mandate borrowers to use the documents for international competitive bidding (ICB) procurement. Model documents have been developed for bidding of goods, works (small and large contracts), and plants (design, build, install), for requesting consulting services proposals, and for prequalification of bidders. Additional things to note about the harmonized standardized bidding documents are as follows:  Best practice contract terms and conditions. The documents contain pro forma contract terms and conditions. In some cases these were developed by the working group, but in other cases external organizations have been engaged with outside industry expertise. For example, contracts of the International Federation of Consulting Engineers are part of some standardized bidding documents. The Federation is internationally recognized as an expert in engineering contracting—using it brings external contracting expertise to the procurement process of the MDBs. It has limitations, though. such documents are criticized as being too engineering oriented and not appropriate for contracting nonengineering systems (information technology systems, for example).108  Extension to country procurement systems. The World Bank and most MDBs mandate borrowers to use the standardized bidding documents for ICB. However, the standardization work often extends to the national level. Mexico, for instance, has a standardized set of bidding documents that are closely modeled on those of the World Bank and IDB, with minor adjustments to suit local conditions, such as language. 109 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT Box 6.3. Harmonization of Procurement Guidelines Over the past 15 years the heads of procurement of multilateral development banks have worked toward harmonization of procurement policies and practices. Starting in about 2006, the procurement guidelines of the Asian Development Bank, IDB, and African Development Bank were revised to more closely align with the World Bank. Except for IDB, whose guidelines were fully aligned with the Bank’s, differences persist. These are largely associated with institutional-specific constraints and regional preferences. A comparison between the Bank’s Guidelines and those of the Asian Development Bank illustrate the nature of the differences. Worldwide Procurement. The Bank requires open bidding regardless of nationality, except where the borrower’s country prohibits commercial relations with another country (boycotts). The Asian Development Bank restricts procurement to firms that are nationals of a member country and goods that are produced in a member country. It does not allow boycotts, except those related to a country’s compliance with its United Nations obligations. Language. For ICB, the Bank allows bidding in any of three languages (English, French, Spanish), whereas the Asian Development Bank works only in English. Advertising. The Bank uses UN Development Business online as the advertising portal. The Asian Development Bank uses its own website as the primary advertising portal. Bidding Methods. The Bank introduced two new procurement methods in the 2011 revisions to its guidelines: framework contracts and a reference to the potential use of country systems. Asian Development Bank guidelines do not include these methods, most likely because it has not caught up with the Bank on these topics. It prescribes special procedures related to emergency and disaster situations, and repeat ordering, which the Bank’s Guidelines do not explicitly address. However, other Bank operational guidelines, outside the procurement guideline, have dealt with emergency fragile situations.a Bidding Procedures: Both organizations use harmonized bidding documents, so procedures are closely aligned. Some differences include:  Asian Development Bank guidelines provide for two-envelope bidding, which the World Bank does not allow.  There are minor differences related to bid security (the Asian Development Bank is more restrictive, allowing only securities provided by a reputable bank).  There are minor technical differences with respect to evaluation of taxes and duties. The domestic preference schemes are virtually identical.  In the case of bid failure caused by prices exceeding the budget, the World Bank requires rebidding. The Asian Development Bank allows negotiation. Oversight: Guideline provisions related to review and no objection are the same for both organizations. Where misprocurement is declared, the Bank requires cancellation of the contract funds from the loan. The Asian Development Bank allows the option of rebidding, depending on the circumstances of the misprocurement. Source: IEG review. a. OP8.5 and then OP8 at the World Bank, and from April 2013, OP 10. 110 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT Although the MDBs, through heads of procurement, have worked to harmonize and standardize bidding documentation, there has been no formal attempt to harmonize their respective procurement guidelines (see Box 6.3). The group is very much a forum for collegial exchange of information on what each organization is doing— with procurement guidelines, it has limited its work to such exchange. In practice, guidelines of other MDBs have become very similar to the Bank’s since 2006 when ADB and IDB, and to a lesser extent the African Development Bank, revised their guidelines. Note that it has been the case that other MDBs align with the Bank, not the other way around. Various reasons have been noted for this. There was a strong lobby from the U.S. Treasury Department for MDBs to issue procurement guidelines identical to the Bank. There was also the push for harmonization initiated by the Second High-Level Forum and the Paris accord. It was reported that there was some resentment among the MDBs who saw alignment to the Bank Guidelines as unilateral. Other notable work of the heads of procurement group includes the following:  Procurement under loans to the private sector. A working group chaired by the Bank produced two guidance notes.109 It was a collaborative effort, including representation from the International Finance Corporation.  Electronic government procurement. This working group departed somewhat from normal heads of procurement group process in that it set up a public website and published much of its outputs (www.mdbegp.org). The working group’s objective was to provide practical resources (a toolkit) to help with the policy, planning, and implementation aspects of introducing electronic procurement. The working group remains active, expanding and keeping resources (the tool kit and website) up to date, supporting clients (case studies, workshops, and so forth), and collaborating with other organizations.  Sanctions. Although not a working group, the group has acted as a venue for the MDBs as they rolled out sanctioning procedures related to fraud and money laundering. Work included contributing to standard definitions of fraud and corruption (now found in MDB procurement guidelines) and addressing the matter of cross-debarment.  Environmentally and socially responsible procurement. A working group was set up in 2007. The ADB Bank working group authored a book on environmentally responsible procurement (ADB 2007), but this is not a 111 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT harmonized document. The working group subsequently disbanded and appears to have dropped harmonized work in this area. COUNTRY SURVEYS—VIEWS OF COUNTRY STAFF AND CLIENTS ON PROCUREMENT HARMONIZATION To what extent is the World Bank able to harmonize in the field? Country surveys found, at one level, a high level of harmonization among policies of the Bank and those of the regional development banks and other major donors, including the African Development Bank, ADB, IDB, JICA, European Bank for Reconstruction and Development, and European Union, depending on the region. Not only are the policies harmonized (as set out in the Guidelines), but those procedures reflected in standardized bidding documents are also largely harmonized. In some instances, the Bank has led the harmonization efforts. In Mexico, the Bank fully harmonized such documents for NCB with IDB. Similarly, the Philippines standardized bidding documents for use on government and donor-funded procurement are harmonized among the Bank, ADB, and JICA. Some policy differences persist, however, and these are problematic in cofinanced projects. As commented on in Bangladesh and Tanzania, it is difficult for executing agencies to deal with procedural and policy differences across a number of donors. It was noted in Ethiopia that European Union regulations sometimes differ from those of the Bank. Bank rules require that projects that have any Bank financial contribution, no matter how minor, are required to follow Bank procurement methods. Although cofinanciers agree to Bank policies (they must), getting waivers of their policies takes time and can delay projects. In Morocco and Senegal, examples were given of the African Development Bank having to seek approval of its board to waive its policy requirement that restricts source of supply to its member countries. The same issue is known to exist with ADB. Parallel financed projects do not face the same policy conflicts, as each financing institution applies its policies to the component it finances—in practice, opting for a parallel financing structure is sometimes done to avoid policy conflicts. However, parallel structures present complications too. In Tanzania, a task team leader gave an example of a project involving five donors, which required the executing agency to apply different policies to different project segments. The Bangladesh survey noted further policy differences. Both JICA and ADB allow two-envelope bidding, contrary to Bank policy. Although ADB is willing to adjust (less on international than on national competitive bidding), JICA is less accommodating, as it sees the Bank’s single envelope process as emphasizing price rather than quality. Additionally, there are differences between Bank and ADB practices for responding to open investigations. Under a cross-debarment 112 CHAPTER 6 BUILDING GLOBAL PRACTICE IN PROCUREMENT agreement, each recognizes the debarment list of the other, but the Bank has an added process related to the Integrity Vice Presidency’s Company Risk Profile Database.110 These different policies and procedures stretch the capacity of implementing agencies that must work with multiple donors. Bilateral agencies are different in that they often conduct procurement themselves, rather than through borrower executing agencies. Harmonization is therefore less feasible, because of the different frames of reference. Government agencies view this as less important as they have little to do with the procurement processes of bilaterals. Overall, the Bank and its development partners have made good progress on harmonization. This is widely recognized to have benefited clients. Persistent differences are problematic. They are often not substantial (two-envelope bidding, for example) yet can hold up projects and stress implementing agency capacity. A simple process to quickly adjust to differences is called for, both for the Bank and its donor partners. 113 Bibliography APEC (Asian Pacific Economic Cooperation). 1999. “Government Procurement Experts Group Non-Binding Principles on Government Procurement”. August Aberdeen Group. 2008. The E-Procurement 1999. Singapore: APEC. Benchmark Report: Driving Year-Over- ASIL (American Society of International Law). Year Superior Performance. Boston: 2013. Will the World Trade Aberdeen Group. Organization Finally Tackle Corruption ADB (Asian Development Bank). 2002. in Public Purchasing? The Revised “Understanding Public Procurement.” Agreement on Government Governance Brief, Asian Development Procurement. Bank. Manila: ADB. http//www.asil.org/insights130415.cf m. ———. 2007. Environmentally Responsible Procurement—A Reference Guide. Manila: Blue, Richard, and John Eriksson. 2011. ADB. “Evaluation of the Implementation of the Paris Declaration: United States AfDB. (African Development Bank). 2011. Government Synthesis Report”. “Evaluation of Paris Declaration Arlington, VA: The United States Implementation at the African Agency for International Development. Development Bank.” Operations Evaluation Department. Tunis: AfDB. Burger, Philippe, and I. Hawkesworth. 2011. How to Attain Value for Money: ———. 2012. “Determination of Financial Comparing PPP and Traditional Thresholds for Modes of Procurement of Infrastructure Public Procurement. OECD Goods, Works and Non-Consulting Journal on Budgeting, Volume 2011/1. Services”. African Development Bank Paris, France: OECD. Group, Procurement Policy and Fiduciary Services Department. January Casartelli, G., E. Wolfstetter. 2007. "Selection and 2012. Tunis: AfDB. Employment of Consultants: the Effectiveness of the World Bank's AfDB, IDB (Inter-American Development Bank), Policy" Working Paper. Washington, World Bank, and Curtin University of DC: World Bank. Technology. 2007. International Survey of Procurement. Perth. Commonwealth of Australia. 2008. Commonwealth Procurement Guidelines, Agaba, Edgar, and Nigel Shipman. 2007l “Public December 2008. Issued under Procurement Reform in Developing Regulation 7 of the Financial Countries: The Uganda Experience.” In Management and Accountability G. Pig and K.V. Thai (eds.) Advancing Regulations, 1997. Canberra: Public Procurement: Practices, Innovation, Commonwealth of Australia. and Knowledge Sharing (Boca Raton, FL: Pr. Academics Press), 373-91. Debevoise, E.W., and C. Yukins. 2010. “Assessing the World Bank’s Proposed Al-Azar, Rima. 2011. “A Tale of Two CDDs: Revision of Its Procurement Lessons Learned for Mitigating the Risk Guidelines,” The Government Contractor, of Fraud and Corruption in CDD 52, (21). Operations.” PowerPoint presentation, World Bank, Washington, DC. Dehn, Jan, Ritva Reinikka, and Jakob Svensson. 2003. “Survey Tools for Assessing Andrews, Matt. 2013. The Limits of Institutional Performance in Service Delivery.” In Reform in Development: Changing Rules Francois Bourguignon and Luiz Pereira for Realistic Solutions. Cambridge: da Silva (eds.), Evaluating the Poverty and Cambridge University Press 128-130. Distributional Impact of Economic Policies. 114 BIBLIOGRAPHY New York: Oxford University Press and ———. 1990. “A Review of Procurement in World Bank. Policy-Based Lending.” Report no. 8341, Washington, DC: World Bank. Dener Cem, Joanna. A Watkins and William. L. Dorotinsky. 2011. “Financial ———. 2002. India Country Assistance Evaluation. Management Information Systems—25 Washington, DC: World Bank Years of World Bank Experience on ———. 2005. The Effectiveness of World Bank What Works and What Doesn’t.” World Support for Community-Based and –Driven Bank, Washington, DC. Development: An OED Evaluation. De Silva, S. 2000. “Community-based Washington, DC: World Bank. Contracting; A Review of Stakeholder ———. 2006a. Debt Relief for the Poorest—An Experience.” Washington DC: World Evaluation Update of the HIPC Initiative. Bank. Washington, DC: World Bank. DfID (Deparment for International ———. 2006b.Engaging with Fragile States: An Development). 2011. “Approach to Evaluation of World Bank Support for Low- Value for Money.” London: Department Income Countries Under Stress. for International Development. Washington, DC: World Bank. European Union, World Bank, and Inter- ———. 2006c. Hazards of Nature, Risks to American Development Bank. 2009. Development – An IEG Evaluation of World Perú Informe de la gestión de las finanzas Bank Assistance for Natural Disasters. públicas Basado en el marco de referencia Washington, DC: World Bank. para la medición del desempeño (PEFA). Lima: European Union, World Bank and ———. 2006d. Small States: Making the Most of Inter-American Development Bank. Development Assistance – A Synthesis of World Bank Evaluation Findings. Gartner Group. 2012. “Survey Shows Why Washington, DC: World Bank. Projects Fail.” http://www.gartner.com/id=2034616. ———. 2007a. The Development Potential of Regional Programs – An Evaluation of GSDRC. 2010. Helpdesk Research Report: Public World Bank Support of Multicountry Financial Management and Frontline Operations. Washington, DC: World Service Delivery. Bank. http://www.gsdrc.org/docs/open/HD 653.pdf,. ———. 2007b. Water Management in Agriculture—Ten Years of World Bank IDB (Inter-American Development Bank). 2010. Assistance, 1994-2004. Washington, DC: “Guide for Acceptance of the Use of World Bank. Country Procurement Systems.” June 2010. Washington DC: IDB. ———. 2008a. Country Financial Accountability Assessments and Country Procurement IAEA (International Atomic Energy Agency). Assessment Reports: How Effective Are 2011. Agency for Financial Regulations World Bank Fiduciary Diagnostics? and Rules AM VI/1: Procurement of Washington, DC: World Bank. Goods and Services, February 16. Mimeo. ———. 2008b. Public Sector Reform: What Works IEG (Independent Evaluation Group). 1977. and Why? An IEG Evaluation of World “The Role and Use of Consultants in Bank Support. Washington, DC: World Bank Group Projects.” Report no. 1824, Bank. Washington, DC: World Bank. ———. 2008c. The World Bank in Indonesia 1999- ———. 1981. “An Interim Report on 2006--Country Assistance Evaluation. Procurement Issues in Bank-Financed Washington, DC: World Bank Projects.” Report no. 3557, Washington, DC: World Bank. 115 BIBLIOGRAPHY ———. 2009a. Bangladesh Country Assistance ———. 2011c. Trust Fund Support for Evaluation 2001-2008. Washington, DC: Development – An Evaluation of the World World Bank Bank’s Trust Fund Portfolio. Washington, DC: World Bank. ———. 2009b. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition ———. 2011d. World Bank Country-Level and Population--An Evaluation of World Engagement on Governance and Bank Group Support since 1997. Anticorruption: An Evaluation of the 2007 Washington, DC: World Bank Strategy & Implementation Plan. Washington, DC: World Bank. ———. 2009c. Review of IDA Internal Controls: An Evaluation of Management’s Assessment ———. 2011e. World Bank Progress in and the IAD Review. Washington, DC: Harmonization and Alignment in Low- World Bank. Income Countries. Washington, DC: World Bank. ———. 2009d. The World Bank Group Guarantee Instruments 1990-2007 –An Independent ———. 2012a. The Matrix System at Work: An Evaluation. Washington, DC: World Evaluation of the World Bank’s Bank Organizational Effectiveness. Washington, DC: World Bank. ———. 2010a. Cambodia – An IEG Country Assistance Evaluation, 1999-2006. ———. 2012b. “The World Bank and Public Washington, DC: World Bank Procurement. An Independent Evaluation.” Approach Paper. ———. 2010b. Climate Change and the World Bank Washington, DC: World Bank. Group. Phase II: The Challenge of Low- Carbon Development. Washington, DC: ———. 2012c. The World Bank Group’s Response to World Bank. the Global Economic Crisis—Phase II. Washington, DC: World Bank. ———. 2010d. IDA Internal Controls: Evaluation of Management’s Remediation Program-- ———. 2013. “Comparison of the International Report on Management’s Implementation Instruments on Public Procurement.” and IAD’s Review of the Five-Point Action Background paper. Washington, DC: Plan. Washington, DC: World Bank. World Bank. ———. 2010d. Mozambique Country Program IMF (International Monetary Fund). 2012. Evaluation. Washington, DC: World Update on the Bank’s Business Bank. Modernization: Results, Openness, and ———. 2010e. Poverty Reduction Support Credits: Accountability Spring 2012.”IMF World An Evaluation of World Bank Support. Bank Development Committee. Washington, DC: World Bank. Kivistö, T., V.M. Virolainen, and E. Tella. 2003. ———. 2010f. The World Bank in Nigeria 1998- Consortia purchasing and logistics in 2007—Country Assistance Evaluation. Kuopio area (Finland)--Lessons learned Washington, DC: World Bank from a 4-year project. IRSPP conference, Budapest, and in KV Thai et al. (eds.), ———. 2011a. “Aid Effectiveness 2005-10: (2005): Benchmarking municipal public Progress in Implementing the Paris procurement activities in Finland. In Declaration.” World Bank, Washington, Challenges in Public Procurement: an DC. International Perspective. ———. 2011b. Capturing Technology for Korpela, J., and M. Tuominen. 1994. Development – An IEG Evaluation of World Benchmarking Logistics Performance with Bank Group Activities in Information and an Application of the Analytical Hierarchy Communication Technologies. Process. Lappeenranta, Finland: Washington, DC: World Bank. Lappeenranta University of Technology. 116 BIBLIOGRAPHY Kruger. K, 2009. “Ban on negotiations in tender ———. 2011c. Strengthening Country Procurement procedures – Undermining the best Systems, prepared for the 4th High Level value for money.” Journal of Public Forum on Aide Effectiveness, Procurement, 4. November 29 – December 1, 2011, Busan, Korea. OECD/DAC Working Mackie, Andrew, and Giovanni Caprio. 2011. Party on Aid Effectiveness Task Force Assessing the Impact of the FEFA on Procurement. Framework. Washington, DC: PEFA Secretariat. OECD-DAC and World Bank. 2004. “Mainstreaming and Strengthening Ministry of Foreign Affairs of Denmark. 2008. Public Procurement: A Strategic “Evaluation of the Implementation of Framework.” Draft report of the 3rd Joint the Paris Declaration: Phase One Round Table on Strengthening Synthesis Report.” Denmark: Ministry Procurement Capacities in Developing of Foreign Affairs. Countries. Washington, DC, and Paris. OECD (Organization for Economic Cooperation Pallas, Christopher L., and Jonathan Wood. and Development). 2003. Mainstreaming 2009. “The World Bank’s Use of the Procurement Function into the Public Expenditure Policy and Effectiveness Country Systems for Procurement: A Dialogue. Paris: OECD. Good Idea Gone Bad?” Development Policy Review 27 (2): 215-230. ———. 2005a. “Paris Declaration on Aid Effectiveness--Ownership, PEFA Secretariat. 2011. Independent Evaluation of the Public Expenditure and Financial Harmonization, Alignment, Results and Accountability Program. Washington, DC: Mutual Accountability.” Paris, France: PEFA. OECD. Pollit, C., and G. Bouckaert. 2004. Public ———. 2005b. “Strengthening Procurement Management Reform: A Comparative Capacities in Developing Countries.” Analysis. New York: Oxford University Volume 3 of the OECD-DAC Guidelines Press. and Reference Series, Harmonising Donor Practices for Effective Aid Delivery. Paris: Sanchez, A. 1997. The World Bank Procurement OECD. Function – Adjusting to Emerging Needs. Washington, DC. World Bank. ———. 2006a. “The Challenge of Capacity Retrieved from: Development: Working Towards Good http://intresources.worldbank.org/INT Practice.” DAC Reference Document, PROCUREMENT/Resources/Procurem Paris. ent-Strategy.pdf. ———. 2006b. “Methodology for Assessment of Singh, J., Dilip R. LImaye, Brian Henderson, and National Procurement Systems.” Paris: Xiaoyu Shi. 2012. “Public Procurement OECD. Of Energy Efficiency Services: Lessons ———. 2009. “Compendium of Country from International Experience.” Examples and Lessons Learned from Washington, DC: World Bank. Applying the Methodology for Tonkin, Christine. 2012. “Procurement Assessment of National Procurement Governance Reform at the International Systems: Sharing Experiences.” OECD Atomic Energy Agency.” Mimeo. Journal on Development 9(4). Thai, K.V., et al. 2005. “Benchmarking Municipal ———. 2011a. International Engagement in Fragile Public Procurement Activities in States: Can’t We Do Better? Paris: OECD. Finland.” In Challenges in Public ———. 2011b. A Practical Guide for Transforming Procurement: An International Perspective. Procurement Systems. Paris: OECD. Academic Press. 117 BIBLIOGRAPHY UNCITRAL (United Nations Commission on ———. 2007a. “Detailed Implementation International Trade Law) Secretariat. Review. India Health Sector. 2006-2007.” (1994, 2011) “UNCITRAL Model Law on INT, Washington, DC: World Bank. Public Procurement.” UNCITRAL, ———. 2007b. “Mandatory Provisions for NCB Vienna, Austria. (for Bank/IDA financed contracts).” Volcker, Paul, Gustavo Gaviria, John Githongo, Washington, DC: World Bank. Ben W. Heineman Jr., Walter Van ———. 2007c. “Operational Approaches and Gerven, and Sir John Vereker. 2007. Financing in Fragile States.” Independent Panel Review of the World Washington, DC. Bank Group Department of Institutional Integrity. Washington, DC: World Bank. ———. 2007d. “Strengthening World Bank Group Engagement on Governance and World Bank. 1997. “The World Bank Anti-Corruption.” Washington, DC: Procurement Function- Adjusting to World Bank. Emerging Needs.” SecM97-854, Washington, DC: World Bank. ———. 2008a. “Enhanced Support to Staff Working in Fragile and Conflict- ———. 2002a. “Fiduciary Management for Affected States.” Washington, DC: Community-Driven Development World Bank. Projects: A Reference Guide.” Washington, DC: World Bank. ———. 2008b. “Use of Country Procurement Systems in Bank-Supported Operations: ———. 2002b. Guidelines for Country Procurement Proposed Piloting Program.” R008- Assessment Reviews. Washington, DC: 0036/5. Washington, DC: World Bank. World Bank. ———. 2009a. “Guidance to Staff for Using the ———. 2003. “Standard Bidding Documents: Company Risk Profile Database Supply and Installation of Information (CRPD).” World Bank, Washington, DC. Systems.” Washington, DC. ———. 2009b. “Handling Suspected and ———. 2004. “Country Procurement Issues Alleged Fraud and Corruption in Paper—Rwanda.” Washington, DC: Procurement: OPCS and INT Protocol.” World Bank. Washington, DC: World Bank. ———. 2005a. “Africa Capacity Building Task ———.2009c. “Piloting Program in Use of Force Report: Forging Effective States Country Systems: First Progress and Engaged Societies.” Washington, Report.” Operations Policy and DC: World Bank. Country Services. Washington, DC: ———.2005b. “Expanding the Use of Country World Bank. Systems in Bank-Supported Operations: ———. 2010a. Financial Management Sector Issues and Proposals.” Operations Strategy FY11-13. 6, 9. Washington, DC: Policy and Country Services, Financial Management Sector Board. Washington, DC. ———. 2010b. “Guidance Note on the ———. 2005c. “Meeting the Challenge of Operational Risk Assessment Africa’s Development: A World Bank Framework (ORAF) – Risks to Group Action Plan.” Africa Region Achieving Results.” Operations Policy Department, Washington, DC: World and Country Services, World Bank, Bank. Washington, DC. ———. 2006. “Quality of Information and ———. 2010c. “Guidance Note: Procurement Communication Technology Arrangements Applicable to Public- Components in Bank Projects.” Quality Private Partnership Contracts Financed Assurance Group Assessment, Washington, DC: World Bank. 118 BIBLIOGRAPHY under World Bank Projects.” Services under IBRD Loans and IDA Washington, DC: World Bank. Credits and Grants by World Bank Borrowers.” Washington, DC: World ———. 2010d. Monitoring Report 2010. PEFA Bank. Secretariat Report. World Bank, Washington, DC. ———. 2011g. “Strengthening Country http://pefa.org/sites/pefa.org/files/att Procurement Systems.” Prepared for the achments/ENG-%20MR-10- 4th High Level Forum on Aide FinalReport-May192011.pdf. Effectiveness, November 29 – December 1, Busan, Korea. Washington, DC: ———. 2010e. “Piloting Program in the Use of World Bank. Country Procurement Systems—Second Progress Report.” Washington, DC: ———. 2011h. “Strengthening World Bank World Bank. Engagement on Governance and Anti- Corruption: Procurement Companion ———. 2010f. “Procurement Risk Assessment Piece.” Washington, DC: World Bank. and Management System User Guide.” OPSOR, May 2010, Updated July 2011, ———. 2011i. World Development Report: Conflict, November 2011, January 2012. Security, and Development. Washington, Washington, DC: World Bank. DC: World Bank. ———2010g. “Public Procurement of Energy ———. 2012a. “Analysis of World Bank Efficiency Services: Lessons from Completed Cases of Fraud and International Experience.” Washington, Corruption from the Perspective of DC: World Bank. Procurement.” Integrity Vice Presidency, Washington, DC: World ———. 2010h. “Republic of Moldova Country Bank. Procurement Assessment Report.” Washington, DC: World Bank. ———. 2012b. “Applying the Principles for Good International Engagement in ———. 2011a. “Community-Driven Fragile States and Situations to Development Projects: A Review of Strengthen and Transform Public Governance Issues.” OPCS Document, Procurement Systems.” Background World Bank, Washington, DC. note, OPSOR. Washington, DC: World ———. 2011b. Country Partnership Strategy Bank. Progress Report for the Republic of Moldova ———. 2012c. “The Bank's Procurement Policies for the Period FY09-FY13. Washington, and Procedures: Initiating Discussion DC: World Bank. and Approach Papers--Proposed ———. 2011c. “Diversity Benchmarking Study Review.” Report submitted to the Audit in Selected Countries.” OPSOR, Mimeo. Committee and to CODE, Washington, Washington, DC: World Bank. DC: World Bank. ———. 2011d. “Financial Management ———. 2012d. “Fiduciary Management for Information Systems—25 Years of Community-Driven Development World Bank Experience on What Works Projects: A Reference Guide.” and What Doesn’t.” Washington, DC: Washington, DC: World Bank. World Bank. ———. 2012e. Financial Management and ———. 2011e. “Green Procurement Policy Procurement in World Bank Operations: Framework in Selected Countries.” Annual Report for FY11. Washington, OPSOR, Mimeo. Washington, DC: DC: World Bank. World Bank. ———. 2012f. “Green Procurement in Selected ———. 2011f. “Guidelines: Procurement of Environmental Policy Framework: Goods, Works, and Non-Consulting Review of the World Bank‘s 119 BIBLIOGRAPHY Procurement Policies and Procedures.” ———. 2012p. “Update on the Bank’s Business Background Paper. Washington, DC: Modernization: Results, Openness, and World Bank. Accountability Spring 2012.” DC2012- 0005. Washington, DC: World Bank. ———. 2012g. “Guidance Note for Design and Management of Procurement ———. 2012q. “Virtual VP Review of the Responsibilities in Community-Driven Procurement Policy Review Proposal-- Development Projects.” Washington, An Approach Paper.” Washington, DC: DC: World Bank. World Bank. ———. 2012h. “Investment Lending Reform: ———.2013a. “Approach for Private Sector Modernizing and Consolidating Procurement in Projects financed by Operational Policies and Procedures.” IBRD/IDA, including Public Private OPCS, Washington, DC: World Bank. Partnerships.” Washington, DC: World Bank. ———. 2012i. “OPCPR Procurement Risk Assessment and Management System ———. 2013b. “Audit of the Bank’s Operational User Guide.” Washington, DC: World Framework for Using Investigation Bank. Results in Bank-Funded Projects.” Report No. IBRD FY12-16. Washington, ———. 2012j. “Procurement and PPP DC: World Bank. Transactions Guidance for MDB Public- Sector Engagements and Procurement ———. 2013c. “Procurement in World Bank Principals Applicable to Private Sector Investment Operations. Phase I: A Transactions—Guidance for MDBs.” Proposed New Framework.” OPCS. Washington, DC: World Bank. Washington, DC: World Bank. ———. 2012k. “Procurement Risk Assessment ———. 2013d. “Red Flags Learning Tool.” and Management System User Guide.” Integrity Vice Presidency. Washington, DC: World Bank. http://go.worldbank.org/D1K227ET81. ———. 2012l. “Project Appraisal Document on a ———. 2013e. “Situations of Urgent Need of Proposed Credit in the Amount of SDR Assistance or Capacity Constraints: 26.1 million ($40.0 million Equivalent to Simplified Procurement Procedures and the Republic of Moldova for the Guiding Principles.” Moldova Education Reform Project).” ———. 2013f. “Supporting Countries in Washington, DC: World Bank. Strengthening their Public Procurement ———.2012m. “Public Procurement of Energy Systems: Second Generation Reforms” Efficient Products: Lessons from (draft). Operations Risk Management Around the World.” Energy Sector Department. Washington, DC: World Management Program. Washington, Bank. DC: World Bank. ———.2013g. “Use of Country Procurement ———.2012n. “Review of the World Bank’s Systems.” Background Paper, Procurement Policies and Procedures: Washington, DC, World Bank. Achieving Social Diversity Through the World Bank Institute. 2012. “Public-Private use of Supplier Preferences in the Partnership Reference Guide, Version Procurement Process.” Background 1.0.” Washington, DC: World Bank. paper. Washington, DC: World Bank. World Trade Organization. 1996. “Agreement ———.2012o. “Review of the World Bank’s on Government Procurement (GPA).” Procurement Policies and Procedures, Geneva: World Trade Organization. The Use and Impact of the Bank’s Policy of Domestic Preferences.” Background Paper. Washington, DC: World Bank. 120 Endnotes Chapter 1 1Program for Results was approved by the Bank’s Board on January 24, 2012. The Bank’s modernization program is described in IMF (2012), which put procurement reform on the agenda for simplification (p. 15). 2An Initiating Memorandum prepared by Bank management was discussed by the Board on February 29, 2012, followed by a discussion of Management’s Phase I proposals on April 12, 2013. 3Looking at goods and civil works alone, the share of competitive bidding in all prior review contracts has averaged around 77 percent over the past decade and has not shown a downward trend (Appendix 1a). 4See World Bank (2013b). In the present evaluation IEG reviews the incorporation of Value for Money in procurement in Volume II, Chapter 1. 5Based on Board discussions, as well as a review of major IEG evaluations. Details are available in IEG’s Approach Paper and summarized in Appendix 1a. for example, IEG’s country program evaluations for Bangladesh, Cambodia, Liberia, 6 See Timor-Leste, and Turkey. 7To some extent, other MDBs also reflect some of these issues, though it appears on a lesser scale, as discussed in IEG’s approach paper. 8Investment lending has consistently formed the core of Bank lending, averaging around 67 percent of new lending by value over most of the last decade and more than 80 percent of the numbers of new loans. 9 Bankprocurement policies are described in its OP/BP 11.00. The Bank’s Articles of Agreement also require the Bank to “ensure that the proceeds of any loan are used only for the purposes for which the loan was granted.” 10See, for example, the CDD Community of Practice Group (http://connect.worldbank.org/explore/SDV/cdd/default.aspx). Concentrated particularly in portfolios such as agriculture, health and education, CDD projects amount to some $2 billion per year and have an outstanding portfolio of around $30 billion. 11Government Procurement Agreement, World Trade Organization, and Model Procurement Law, the United Nations Commission on International and Trade Law. Many caveats apply, however, in comparing the Bank procurement system to external models. 12 Upper limit, assuming all prior review contracts are wholly Bank financed. 13Government clients included the Bank’s counterparts in the Ministry of Finance, the national procurement authority, and staff in ministries executing Bank projects. to some substitution in the case of field visits. Tanzania was substituted for 14 Leading Ghana, and Mexico and Peru were substituted for Colombia and Argentina. 15The field visit countries were: Europe and Central Asia: Azerbaijan and Turkey; Middle East and North Africa: Morocco; South Asia: Bangladesh; Africa: Ethiopia, Senegal, 121 ENDNOTES Tanzania; East Asia and Pacific: Indonesia, the Philippines; and Latin America and the Caribbean: Mexico and Peru. In addition the following countries were chosen for desk review: Albania, Bhutan, Colombia, Ghana, Honduras, Lao PDR, Mozambique, Rwanda, Sierra Leone and Vietnam. Chapter 2 16All WB country strategy reports approved between FY2002 and FY2011: CAS, Country Partnership Strategy, and Interim Strategy Note reports. 17Of the 124 countries whose CAS reports were available for review, 3 countries had 4 reports prepared during the evaluation period, 33 had three reports, 68 had 2 reports, and 20 had only one report. 18The 13 CASs that did not reference procurement were from Bulgaria, the Central African Republic, Comoros, Kiribati, Macedonia FYR, Namibia, Poland, Romania, Saõ Tome and Principe, Somalia, Sudan, Swaziland, and Zimbabwe,. Some among these include countries that had undertaken MAPS exercises (Poland), or that were included in the Bank’s use of country systems pilot (Macedonia). 19All World Bank Country Assistance Strategy documents are available on the individual country pages of the Bank’s website (http://www.worldbank.org). Completed CPARs are also publicly disclosed with permission from the client country. 20Some CPARs in later years integrated the MAPS diagnostic tool, although for purposes of this study they are counted as CPARs. These reports were not very numerous; of the sample of 40 CPARs reviewed, only 6 had the MAPS tool embedded in their scope. The CAS reports that made mention of PEFA assessments included; Botswana (2009), Brazil (2008), Tonga (2011), and Uganda (2010). 21 Examples include Tajikistan (2006), Panama (2006), and the Republic of Yemen (2009). 22Including El Salvador (2010), Guatemala (2005), Tanzania (2007), Ukraine (2008), and India (2009). 23 Broadly following the four pillars of the MAPS framework, discussed later in this chapter. 24Given that the legislative area formed a considerable part of the focus of CPARs, as discussed later in this chapter, this is a somewhat surprising finding, and suggests that there was an awareness of limitations of legal reform on its own. 25See Appendix 2a Table 7 for the list of CAS reports that included Bank support for the procurement institutional framework. 26Examples include Argentina (2006), Bangladesh (2006), Colombia (2008), Côte d’Ivoire (2010), and Lebanon (2006). 27With the caveat that the analysis of time series data is based on a limited number of observations per year. This is one reason why three-year moving averages were used for the analysis. 28The most recent CPAR was undertaken by the Asian Development Bank; the Bank did, however, participate. 122 ENDNOTES 29 Time series data are not available for most regions. Chapter 3 30Box 3.1 describes previous and related IEG evaluations of the Bank’s procurement diagnostics. For this review, IEG surveyed 37 CPARs and three Country Procurement Issues Papers, 31 which are an abbreviated version of the CPAR. 32Including the Stage I analysis of the Bank’s use of country systems piloting program, where a total of fourteen MAPS assessments were conducted, nine of which IEG reviewed for this report (Chapter 3). 33The Mozambique 2002 CPAR, for instance, featured a section on the registration of importers and import licensing, a subject that would not have received such attention in a purely MAPS assessment. 34Details of relevant PEFA indicators are given in Appendix tables C.4–C.6. Blends of these instruments have also been devised, including Public Expenditure and Institutional Reviews, and integrated assessments (CIFA, Public Expenditure Management and Financial Accountability Review) that combine elements of PER, Country Financial Accountability Assessment, and/or CPAR. 35IEG reviewed all CPARs and MAPS conducted as part of the use of country systems piloting program, for the 11 case study and ten desk review countries. IEG also included results from a further eight reports from seven additional countries, to add, for example, a range of fragile and conflict-affected countries, not included in the original sample The template of questions is scored (like MAPS) on a scale of 0 to 3. 36Moreover, the Bank, through the WBI, had other instruments targeted more specifically at civil society engagement, as discussed further in this report (Volume I Chapter 5). 37The half-day roundtable with about 30 senior government officials and donor representatives organized by the Ministry of Economic Development for the Azerbaijan 2009 CPAR was a typical forum, providing “an excellent opportunity for discussion and exchange of views.” 38Although other donors have utilized the MAPS tool, there was little collaboration on the specific Bank-led reports surveyed by IEG, undertaken largely in the context of the Bank’s use of country systems pilot, as the use of country systems piloting program was geared specifically to the World Bank. 39This is in line with the findings of the earlier IEG study on fiduciary diagnostics (IEG (2008a), which noted, “[Country Financial Accountability Assessments] and CPARs evaluated lacked a satisfactory framework for risk analysis, thereby preventing the Bank from arriving at a comprehensive risk rating which the Guidelines for these diagnostic instruments require. This has limited their relevance for the fiduciary goal.” 40However, it does include, as a possibility in the suggested outline for MAPS reports, “A section on the assessment of outstanding weaknesses in the procurement system, classifying them into the high, medium and low categories with regard to the risk such weaknesses may pose to the system and suggestions as to how to keep these risks at an acceptably low 123 ENDNOTES level. Such suggestions may form the basis for a prioritized reform strategy intended to address identified weaknesses.” 41 With an average score of 2.7 out of 3. 42This usually starts with the country’s specific public procurement law. In the absence of a public procurement law, as in 2002 Bangladesh and Mozambique, other applicable instruments such as an old compilation of financial rules and a government decree, respectively, are discussed. In-depth discussion of all other areas with no mention of accessibility was common. This 43 was the case for the Morocco 2000 CPAR and 2008 MAPS, the Sierra Leone 2004 CPAR, Turkey 2001 CPAR, and Vietnam 2002 CPAR. 44The absence of standard bidding documents was noted with concern in CPARs in Colombia 2001 and 2005, Honduras 2005, Mexico 2007, Peru 2001 and 2005, and Vietnam 2002. 45The use of country systems piloting program and its bidding documents analysis is explored in more detail in Volume II, Chapter 1. 46See Volume II Chapter 4. 47The Cambodia 2004, Lao PDR 2002, and Nepal 2001 CPARs noted low capacity but did not analyze the system or contain constructive discussion of reform in this area. 48Cambodia 2004, Colombia 2001 and 2005, Mozambique 2002, Pakistan 2000, and Sierra Leone 2004 CPARs mention this specifically. 49 This was the case in both Ethiopia CPARs (2002 and 2012). The 2005 Honduras CPAR was another example of scant coverage of enforcement, stating only that there are “generally applicable legal penalties for corruption.” And the Philippines 2003 CPAR discussed governmental and non-governmental agencies that address corruption in depth, but with little idea of implementation. 50The CPAR asks only, “Are procurement evaluations/audits conducted? If so, describe scope, frequency, who carries them out, etc.” MAPS indicator 9 and its five subindicators deal with audit. 51The six NLTA projects were selected after conducting a search of all NLTAs approved between FY02 and FY11 financed by Bank Budget and two trust funds: the Governance Partnership Facility and the Bank-Netherlands Partnership Program. A word search for procurement (and related key words; for example, bid, tender, acquisition) in the title of the NLTA/Trust Fund gave a total sample of 60 “projects.” Of these, 30 were funded by Bank Budget, and for 9 of these, there was documentation available. Of these, three focused on land acquisition, leaving a sample of six focused on themes relevant to the public procurement framework. 52In some respects, the sample differs from the sample of reports reviewed for quality. Because of the need to have two or more reports over time, countries where there was only one report were dropped (Appendix Table C.2). 124 ENDNOTES 53Because PEFAs have a distinct focus and coverage and are more integrated with budget management issues, IEG also undertook a separate, complementary analysis of PEFAs and their findings relevant to building procurement capacity. 54Four (high) and three (substantial) indicate that recommendations were implemented and objectives achieved with few major deficiencies; two (modest) and one (low) denotes progress with several major deficiencies. 55The procurement authority has worked closely with the Ghana Institute for Management and Public Administration and other training institutions to offer a number of medium- and long-term procurement courses 56In 2007 and 2008, in Ghana, the PPA implemented a comprehensive short-term training program aimed at disseminating the Public Procurement Act. Based on 25 targeted training modules, 3,840 procurement practitioners were trained, 4,222 members of Tender Committees and Tender Review Boards, 200 service providers, and 353 staff of oversight institutions, totaling 8,615 people. In addition, with the support of a World Bank-funded IDF grant, 227 internal audit staff was trained in audit of procurement and assets management systems. More training is being developed for contract management and oversight institutions. A 2006 report on establishment of a career path and career development of procurement professionals and practitioners in the public/civil service was approved by the Office of the Head of Civil Service and is presently awaiting formal approval by the Public Service Commission. 57Although IEG reviewed diagnostic reports for 18 countries, it was not possible to score them for each indicator. So discussions cover only those countries whose reports have information on the indicator of interest. 58“Access to information, especially on public contracts, is remarkable in Morocco: public procurement electronic portal, publication of statutory instruments, online publication of invitation for bids, award decisions, procurement plans, list of firms excluded from public procurement, bidding documents (can be downloaded), information on firms registered (free of charge) for access by email on procurement notices in their field of interest, etc… are the typical features of the procurement electronic portal. This system is connected with and fed in by the government’s public procurement management information system. Lastly, the information system is accessible countrywide, including in rural municipalities thanks to information technology connection of the GTK’s fiscal agents in the municipalities” (Morocco MAPS 2010). 59Although there is a high level of compliance (94 percent) to the presence of agency websites that provide minimum, up-to-date procurement information there is only 41 percent compliance rate for the preparation of PMRs among the pilot agencies and a lower rate of 18 percent for the posting of these reports at the agency website. “has been developing a very sound, productive and inclusive planning and budgeting 60It process, with Cabinet participation in ceiling-setting, under an MTEF framework which has played an important role in maintaining fiscal discipline over this period and in determining the strategic allocation of resources.” Annual budgets are prepared by the Ministry of Finance with the involvement and the active participation of line ministries, departments and agencies and subnational governments, and even line managers, who 125 ENDNOTES assess the procurement needs under existing and new contracts, and recurrent needs (Rwanda PEFA 2008). 61Drawn from a larger sample of 29 countries that had repeat PEFA assessments carried out in 2011 or before, and where the repeat assessment had participation by the Bank as team leader or member. Countries selected for review are those described in IEG’s country selectivity framework described in Chapter 1. However, in some dimensions, countries did not have indicators that could be reasonably compared because of new evidence in the repeat assessment that was not available to the previous assessment, or because no scores were given for one of the assessments, there was questioning of earlier scores, definition changes, different interpretations, or different sampling. 62This result includes both changes directly related to procurement as measured by PI-19, and changes in other public financial management indicators that are linked to procurement in ways discussed in Box 3.5. PI-4 is excluded, as it is an overall outcome indicator rather than a process indicator. 63 Repeat assessments used the original formulation of PI-19 except where noted. 64 This could also be characterized as centralized and decentralized. 65Repeat assessments considered were carried out an average of 2.7 years after the previous assessment, ranging from 1 to 4 years. Seven countries carried out repeat assessments less than 2 1/2 years after the previous assessment, for an average 1.8 years. This is a shorter interval than the recommended guideline of guideline of 3– 5 years, set because of the normal time needed for reforms to demonstrate measurable results. One consideration to keep in mind is that the gap between the dates of the two assessments may not necessarily reflect the gap in the underlying datasets being used. In the Kenya case (gap between missions 2.5 years), for example, the first assessment used data for the 2004/2005 fiscal year, while the repeat assessment used data for the 2007/2008 fiscal year; thus the three year recommended waiting period between assessments was met (World Bank 2010d). 66However, at a disaggregated level, there are high or improving scores even in the functional categories, indicating that even in the most difficult areas, improvement is possible. 67However, countries need to be selective in choosing areas for reform. Cross-country research shows that even OECD countries may have low PEFA scores in some areas (see Andrews 2013). 68 Drawing on IEG case studies, as well as on Mackie and Caprio (2011). 69 No PEFA assessment has yet been carried out for Mexico. Chapter 4 70As illustrated in IEG’s field visits, Mexico was supported by DPL lending hands-on knowledge sharing by procurement unit staff in the context of projects. In Ethiopia, the recommendations and action plan of the 2002 CPAR were supported by procurement- focused components of two Bank projects. Phased implementation of the latest 2012 CPAR will be supported by elements of a multi-sector project for basic services. In Morocco, the action plan proposed by the first CPAR of 1999 was supported by an IDF Grant. In Senegal, 126 ENDNOTES drafts laws and regulations implementing the 2003 CPAR were grant financed by the Bank and a new procurement code was adopted in 2007. In Indonesia, the Bank also promoted implementation of the CPAR and MAPS through grant-financed interventions, some investment lending, and to a large extent, DPLs. 71It is however recognized that even a low number of conditions could still represent important reforms, in some instances. IEG did not however attempt to construct a scale of weights for importance due to the subjectivity this would imply. 72Appendix 2d explains the estimation of “procurement content” in individual operations, and, based on this, IEG’s sample construction for detailed analysis. 73Apart from dedicated procurement reform support in Bangladesh and Bhutan, another eleven investment loans, averaging $19 million in IBRD/IDA commitments, provided technical assistance for procurement reform, along with other elements of public policy support. 74IEG’s desk review also showed that there was a sharp difference between DPOs and IDFs on the question whether a project focused on enhancing sustainable capacity/cadres for public procurement, with 93 percent negative responses for DPOs and in contrast to 93 percent positive responses for IDFs. This suggests some comparative advantage of IDF grants for capacity building. 75In Lao PDR, the World Bank program consisted of two IDF grants (only for procurement), an IDA adjustment credit and an accompanying IDA technical assistance investment loan. The latter two were broad spectrum operations with minor procurement components, in contrast to the more focused dedicated IDF grant. In Bangladesh, issues raised in two dedicated investment loans were supported, to some degree, in four parallel DPLs. 76Of 21 focus countries in the extended IEG sample. Of the 68 DPOs in the sample, 63 were programmatic and 59 were part of a series of three or more DPOs. 77Sample construction is described in Appendix 2d. Templates were completed for 90 projects; the remaining 8 were repeaters in series of programmatic operations that were virtually indistinguishable from previous projects in the series. 78 Legislative and regulatory areas, institutions, markets, and transparency issues. 79Seventy-seven percent of reviewed projects were rated 0 or 1, on a scale of 0–3, in this regard. Scores were based on an analysis of Project Appraisal Documents. 80In the sense that there were few or no signs of procurement being considered actively as input to/part of public financial management. The procurement components of DPLs were sometimes there as components in their own right. 81As pointed out in IEG’s report on the global crisis (IEG 2012c) in many countries (including Peru), that were only moderately affected, crisis lending often took the form of consolidation of medium-term fiscal reform. 82A third of all projects sought to strengthen the quality of the overall legislative framework. This number is pulled down by IDFs and ILs. For DPLs, there is a higher proportion: 43 percent. But only 13 percent of IDFs and 13 percent of investment loans supported legislative reform. 127 ENDNOTES 83Note that some operations were still ongoing and in other cases the results result reporting may not necessarily have covered all aspects of IEG’s analytical template. Bangladesh represents one example where there has been strong focus on e-procurement. 84 There is a further discussion of e-procurement outcomes in Chapter 4. 85Overall, 20 percent of the 90 assessed projects sought to help introduce or strengthen e- procurement However, it is interesting to note this this comprises 13 percent of all DPLs, 40 percent of all IDFs and 15 percent of investment loans. Thus most of the support in this area was from the relatively small IDF instruments. 86There is also a possibility that available documents do not fully cover the aspects under review. 87 This may to some extent be an issue of information in the available documentation. 88Mexico provides an example of strong government commitment - it was the joint efforts of the government, the Bank, and the IDB that led to change. 89 See Volume II, Chapter 1 for a description of the Bank’s pilot. 90 Scores on a scale of 1–4, in the country case study template (see Chapter 1). Chapter 5 91As reaffirmed, for example, at the 4th High Level Forum on Aid Effectiveness in 2011, in Busan, Republic of Korea. IEG’s analysis is based primarily on semistructured IEG interviews with 43 Bank staff and consultants. Interviews focused on the Public Sector Management and Governance group, of the PREM Vice Presidency, OPCS, and operational risk management groups, and staff mapped to the Public Sector, Procurement and Financial Management Network. 92For example, the financial management practice in Europe and Central Asia committed to “following a more consolidated and coordinated approach to public financial management work by building on partnerships with procurement and PREM colleague.” In India: “The PR, financial management, and PREM teams created a [public financial management] group to coordinate dialogue on the PFM agenda.” “(The Financial Management Sector) will work with PREM and Procurement to build country [public financial management] teams to support implementation and avoid duplication through better-coordinated Bank efforts.” “Better coordination among the different committees and working groups of the Sector Board, and among the Anchor, Regions, and loan department (LOA) is needed, as is easier and more accurate reporting on activities. The Sector also will work with PREM and the World Bank Institute to develop [public financial management] training for both financial management and PREM Public Sector Group staff.” These are described in World Bank 2012e and 2010a. 93While retaining similar measures on extent of competition and use of a complaints mechanism, there were added measures of the legal and regulatory framework for procurement, and public access to procurement information. 94OECD experience finds evidence of heightened discourse on financial management, procurement and other public sector management reforms, and better tracking of inputs and outputs. However there is scope for progress, with little evidence that the considerable 128 ENDNOTES cost of the reforms is justified by the benefits achieved (Pollitt and Bouckaert 2004). Some analysis on developing countries is offered in, for example, GSDRC (2010) and Dehn, Reinikka, and Svensson (2003). 95 WBI has also contributed to procurement in the context of good practice for PPP contracts. 96See also the discussion in Volume II, Chapter 1, on contract management, which discusses potential for expansion of the Bank’s role in this regard. 97 As discussed in Volume II, Chapter 4, on efficiency. 98ANSA supported a 2012 World Bank public financial management event, which included procurement. 99Although definitive findings are not available to IEG, early results correlate local capacity with poverty, and also point out how in such circumstances greater use of e-procurement could generate information on procurement performance. Chapter 6 100See OECD-DAC (2005b). This document reported on the work of the Roundtable as it wound down prior to the Paris High Level Forum. It paved the way for the Procurement Joint Venture, set up under Working Party on Aid Effectiveness as a result of the Paris Accord. The Johannesburg Declaration was agreed at the end of 2004 by the roundtable participants, in preparation for the Paris High-Level Forum of 2005. 101The Working Party on Aid Effectiveness was set up in 2003 in the context of the international consensus reached on the actions needed to promote a global partnership for development and accelerate progress toward the Millennium Development Goals. 102The Joint Venture went on to became the Procurement Task Force following the third High Level Form For Aid Effectiveness, held in Accra, Ghana, in September 2008. It continued to function as the Procurement Task Force until the Fourth High Level Forum, held in Busan, Korea, in December 2011. At Busan, it was agreed that the Paris work streams, including the Working Party on Aid Effectiveness and the Task Force on Procurement would continue for six months, by which the new (Busan) structures were to be in place. See www.gngp.org, an ambitious proposal by Peru (Cusco declaration) supported by 103 Morocco, the Philippines, Mongolia, and several other partner countries, supporting a partner country-led network to start regional networks sharing experiences and lessons in procurement reforms. Initially donor members of the old task force would support and advise the new network as a Busan initiative. However, a review of the website of this network indicates that it is not currently active. 104See OECD (2011c). This document provides background on the work of the Roundtable, Procurement Joint Venture, and Task Force on Procurement, describes MAPS, and provides several country case studies. 105OECD-DAC Task Force for Procurement: http://www.oecd.org/document/59/0,3343,en_2649_3236398_43440827_1_1_1_1,00.html 129 ENDNOTES It provides links to the Methodology for Assessment of National Procurement Systems (MAPS) and former baseline indicators and CPIs. This website appeared inactive in 2012. For example, the African Development Bank reports that more than half its client 106 countries have undertaken an assessment of their procurement systems using MAPS. The AfDB uses MAPS in pursuit of its use of country systems policy. 107Procurement Capacity Development Centre: http://www.unpcdc.org. This center is a joint initiative of UNDP and Danida. It carries on the procurement capacity development work, identified by the Roundtable as one of its three core activities. 108 The International Federation produces a range of contract forms. The MDBs contracted the Federation to develop MDB-specific contracts that are then made available to borrowers free of charge. Federation-developed contracts are integral to the standard bidding documents for procurement of works (large contracts). Other such documents (for procurement of goods, for example) use other forms of contract. Reference is made to standard bidding documents for procurement of information technology systems. These provide a form of contract specially designed for information technology, but they were criticized as being engineering oriented, likely reflected that they were derived from the other standard bidding documents and influenced by them. 109Procurement & PPP Transactions Guidance for MDB Public-Sector Engagements (February 2012) and Procurement Principals Applicable to Private Sector Transactions— Guidance for MDBs (June 2012). 110 Bank staff are reluctant to short list firms where there may be an ongoing case under investigation. A firm noted in The Integrity Vice Presidency’s Company Risk Profile Database as under investigation by the Bank may still receive a contract award, but if a suspension or debarment is later imposed in connection with that award, the contract must be cancelled. For the Asian Development Bank, firms are debarred from the date of decision to debar. In theory, the same is true at the Bank, but there is a reluctance to award contracts to firms listed on the company risk database, yet there is no basis to exclude them, so the process is put on hold. Under ADB procedures, a contract signed before a negative finding, even if the firm is under investigation at the time, remains valid. 130 The World Bank and Public Procurement—An Independent Evaluation Volume II: Achieving Development Effectiveness through Procurement in Bank Financial Assistance © 2014 Independent Evaluation Group The World Bank Group 1818 H Street NW, Washington DC 20433 Telephone: 202-458-4497; Internet: www.ieg.worldbankgroup.org E-mail: ieg@worldbank.org Some rights reserved 1 2 3 4 17 16 15 14 This work is a product of the staff of the Independent Evaluation Group (IEG). Note that IEG and the World Bank does not necessarily own each component of the content included in the work. IEG and the World Bank therefore do not warrant that the use of the content contained in the work will not infringe on the rights of third parties. The risk of claims resulting from such infringement rests solely with you. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of IEG, the World Bank, its Board of Executive Directors, or the governments they represent. IEG and the World Bank Group do not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of IEG and the World Bank Group concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of IEG and the World Bank Group, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 Unported license (CC BY 3.0) http://creativecommons.org/licenses/by/3.0. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: IEG (Independent Evaluation Group). 2014. The World Bank and Public Procurement—An Independent Evaluation. Volume II: Achieving Development Effectiveness through Procurement in Bank Financial Assistance. Washington, DC: World Bank. License: Creative Commons Attribution CC BY 3.0 Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by the Independent Evaluation Group or the World Bank Group and should not be considered an official World Bank Group translation. IEG and the World Bank Group shall not be liable for any content or error in this translation. All queries on rights and licenses should be addressed to the IEG, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-3125; e-mail: ieg@worldbank.org. ISBN-13: 978-1-60244-237-5 ISBN-10: 1-60244-237-1 ii CONTENTS Contents ABBREVIATIONS .................................................................................................................................................................. VII ACKNOWLEDGMENTS .......................................................................................................................................................... IX 1. BANK PROCUREMENT AND DEVELOPMENT IMPACT ....................................................................................... 1 Bank Procurement Guidelines: Overall Perceptions and Specific Concerns ............................................................ 4 Bank Procurement Processes ................................................................................................................................... 8 Using Country Procurement Systems ..................................................................................................................... 10 The Bank’s Use of Country Systems Pilot ........................................................................................................................... 10 IEG Analysis of the Bank’s Use of Country Systems Pilot .................................................................................................. 13 Use of Country Systems – the Experience of Other Donors ............................................................................................... 19 Use of Country Systems—Stakeholders’ Views on the World Bank Pilot ........................................................................... 23 Use of Country Systems—Stakeholders’ Views on the Adoption of Country Systems ....................................................... 24 Upstream and Downstream Involvement in Procurement ....................................................................................... 27 Adequacy of Upstream Involvement .................................................................................................................................... 27 Downstream Involvement—Role in Contract Management ................................................................................................ 28 Developing Domestic Suppliers and Offering Domestic Preferences ..................................................................... 30 Present Provisions for Domestic Preferences ..................................................................................................................... 30 Broader Interpretation of Developing Domestic Supply Capability...................................................................................... 31 Environmentally Responsible Procurement—Potential for Bank Use ..................................................................... 33 Bank Policies, Practice, and Experience ............................................................................................................................. 34 International Perspectives ................................................................................................................................................... 35 Achieving Value for Money in Bank Procurement ................................................................................................... 37 Applying Value for Money in the World Bank ...................................................................................................................... 38 2. ADAPTING TO EVOLVING NEEDS....................................................................................................................... 41 Procurement in Community Driven Development Projects ..................................................................................... 41 The CDD Approach and the CDD Portfolio ......................................................................................................................... 42 CDD Lending and Bank Procurement Policies .................................................................................................................... 43 IEG Portfolio Analysis—Performance of All Projects with CDD Elements .......................................................................... 44 IEG Project Reviews—Findings from a Subset of Projects with Community Control ......................................................... 45 Procurement in Fragile and Conflict Situations ....................................................................................................... 50 Analysis of Procurement Issues in FCS—Issues Identified in CASs .................................................................................. 51 Analysis of Procurement Issues in FCS – Review of Project Documents ........................................................................... 54 Analysis of Procurement Issues in FCSs—Staff Questionnaire .......................................................................................... 55 Procurement of Information and Communications Technology .............................................................................. 59 The Bank’s ICT Portfolio and the Scope of the Evaluation ................................................................................................. 59 ICT and Bank Procurement Procedures.............................................................................................................................. 61 IEG Structured Survey: Procurement Staff and Task Team Leaders in a Sample of ICT Projects ................................... 62 IEG Supplementary Interviews: ICT Experts, Procurement Legal Experts ......................................................................... 67 IEG Country Survey Findings—Experience with ICT Procurement .................................................................................... 68 Conclusions ......................................................................................................................................................................... 70 Procurement in Public-Private Partnership Investments ......................................................................................... 72 Public-Private Partnerships in the Bank’s Portfolio ............................................................................................................. 74 IEG Review of Experience in Bank Operations ................................................................................................................... 77 Conclusions ......................................................................................................................................................................... 81 3. MANAGING RISK AND INCREASING TRANSPARENCY ................................................................................... 86 iii Introduction ............................................................................................................................................................. 88 Managing Procurement Risk—Prior Reviews and Thresholds ............................................................................... 88 Concept and Use of Thresholds .......................................................................................................................................... 88 Perceptions of Thresholds—IEG Field Visits ...................................................................................................................... 92 IEG Analysis of Thresholds and Risk Management ............................................................................................................ 95 Results ................................................................................................................................................................................. 95 Conclusions and Implications .............................................................................................................................................. 98 Managing Procurement Risk—Post-Procurement Review ..................................................................................... 98 Managing Procurement Risk—The P-RAMS Instrument ...................................................................................... 102 Managing Risk Before P-RAMS ........................................................................................................................................ 102 The Introduction and Contribution of P-RAMS .................................................................................................................. 107 Perceptions of P-RAMs and ORAF: IEG Field Visits ....................................................................................................... 110 Empirical Analysis of P-RAMS Data .................................................................................................................................. 110 Additional Observations on P-RAMS ................................................................................................................................ 113 Measuring Risk Failure—Misprocurement, Complaints, and Investigations ......................................................... 114 Misprocurement and Procurement Complaints ................................................................................................................. 114 Fraud, Corruption, and the Role of INT ............................................................................................................................. 115 Sanctions, Suspensions, and Corporate Risk ................................................................................................................... 119 Perceptions of the Role of INT—IEG Field Visits .............................................................................................................. 122 4. ACHIEVING EFFICIENCY IN BANK PROCUREMENT ....................................................................................... 124 Procurement Efficiency and Value for Money—Tracking Procurement ................................................................ 124 Main findings ..................................................................................................................................................................... 124 Tracking Contract-Level Procurement Data ...................................................................................................................... 125 Bank-wide Tracking of Procurement: Form 384 ................................................................................................................ 128 Existing Regional Systems ................................................................................................................................................ 130 Future Directions ............................................................................................................................................................... 132 Measuring Procurement Efficiency—IEG Data on Sample Contracts .................................................................. 134 Analysis and Findings: Average Elapsed Times ............................................................................................................... 136 Analysis and Findings: Variation in Elapsed Times—Frequency Distributions ................................................................. 138 Time Taken to Process a Contract—Findings from Regression Analysis......................................................................... 140 Efficiency of Procurement in Africa – IEG Analysis of PROCYS data .............................................................................. 141 Further New Directions for Increasing Efficiency .................................................................................................. 143 Bank Involvement in the Development of E-Procurement ................................................................................................ 146 BIBLIOGRAPHY .................................................................................................................................................................. 151 ENDNOTES .......................................................................................................................................................................... 159 Boxes Box 1.1. Management Review of Bank Procurement Systems ............................................................... 2 Box 1.2. Procurement—Overall Perceptions of Bank Guidelines and Processes ................................... 4 Box 1.3. Procurement Processes—Examples of Inflexibility and Loss of Development Effectiveness ... 9 Box 1.4. Stage II Equivalence Analysis – Bidding Documents Review ..................................................18 Box 1.5. Use of Country Systems—The Bank and the Larger Development Community ......................20 Box 1.6. Using Country Systems—Benefits Perceived by Client Implementing Agencies .....................25 Box 1.7. Using Country Systems—Drawbacks Perceived by the Private Sector ...................................25 Box 1.8. Perspectives on Domestic Preference .....................................................................................32 Box 1.9. Procurement and Energy-Efficient Products and Services ......................................................35 Box 1.10. Asian Development Bank—Loan Agreement Provisions on the Environment .......................36 iv CONTENTS Box 1.11. Incorporation of Maintenance Costs and Life Cycle Costing in Bank Procurement............... 39 Box 2.1. Afghanistan: Emergency National Solidarity Project II ............................................................ 48 Box 2.2. Procurement Issues in the Maranhão Integrated Program...................................................... 49 Box 2.3. Procurement Issues and Procurement Support in FCS—Evidence from Selected CASs ....... 52 Box 2.4. Procurement Guidelines under OP 8.0 and OP 10.0 .............................................................. 52 Box 2.5. Two-Stage and Single-Stage Bidding ..................................................................................... 62 Box 2.6. IEG Review of ICT Procurement—Project Support Provided .................................................. 64 Box 2.7. Messages from the ICT Portfolio Review ................................................................................ 67 Box 2.8. Negotiation to Procure ICT Systems ....................................................................................... 68 Box 2.9. Using Sole-Source Procurement for a System Upgrade ......................................................... 70 Box 2.10. PPP—Missed Opportunities in Other Countries? .................................................................. 80 Box 2.11. India—A Tale of Two PPPs................................................................................................... 82 Box 3.1. Procurement Risk Management in Recent IEG Evaluations ................................................... 86 Box 3.2. The Bank’s Taxonomy of Procurement Risk ........................................................................... 89 Box 3.3. Setting Method Thresholds to Optimize Competition at the AfDB ........................................... 93 Box 3.4. The Procurement Risk Assessment and Management System ............................................ 103 Box 3.5. Integrating Procurement and Project Risk—Tanzania Second Transport Corridor Project ... 107 Box 3.6. Perceptions on New Risk Management Tools—P-RAMS and ORAF ................................... 111 Box 3.7. The Office of Sanctions and Debarment—Two-Tier Sanctions System ................................ 120 Box 4.1. Benchmarking Municipal Procurement Activities—An Example from Finland ...................... 126 Box 4.2. Procurement Tracking Systems at the World Bank—Early Efforts ........................................ 128 Box 4.3. Procurement, Disbursement, Contract Accounting, and Form 384 ....................................... 129 Box 4.4. Proposed New Systems Architecture: The Integrated Procurement Plan ............................. 133 Box 4.5. New Procurement Methods—Framework Agreements and Reverse Autions ...................... 144 Box 4.6. E-Procurement ...................................................................................................................... 145 Tables Table 1.1. Summary of Results of the Bank’s Use of Country Systems Pilot (2005–10) ....................... 12 Table 1.2. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators.................. 14 Table 1.3. Bank Requirements for NCB and Derogations in Select Countries ...................................... 16 Table 1.4. Bank Support During Procurement Design and Planning..................................................... 28 Table 1.5. Contract Management and Implementation ......................................................................... 29 Table 2.1. CDD Projects: ISR Procurement Performance Ratings and Overall Outcomes ................... 46 Table 2.2. IEG Review of ICT Procurement—Project Characteristics ................................................... 63 Table 2.3. IEG Review of ICT Procurement—Reported Use of Bank Procurement Methods .............. 65 Table 2.4. IEG Sample of Bank PPP Projects Reviewed (by region FY04–12)..................................... 78 Table 2.5. Financing Structure of Bank PPP Projects ........................................................................... 79 Table 3.1. Prior Review, Procurement Method and Clearance Thresholds ........................................... 90 Table 3.2. ICB Prior Review Contracts for Goods and Civil Works (numbers and value) (FY00–12) ... 91 Table 3.3. Prior Review Procurement Thresholds: Correlations with Indices of Governance................ 96 Table 3.4. Share of Contracts Designated for Post Procurement Review (2002–10)............................ 99 Table 3.5. Risk Categories and Individual Risk Identification in Pre-P-RAMS PADs by Period .......... 105 Table 3.6. P-RAMS Compliance Rates by Region .............................................................................. 109 Table 3.7. Data on INT Investigation Cases by Region Conducted in FY09–12 ................................. 116 v Table 3.8. Correlation of INT Cases with Project, Procurement, and Financial Management Risk......118 Table 4.1. Summary of System Contributions to Tracking Procurement Performance ........................131 Table 4.2. “No-Objection” Steps for Prior-Reviewed Contracts............................................................135 Table 4.3. Average Elapsed Time between Steps in the Procurement Process (days) .......................137 Table 4.4. Average Elapsed Time and Contract Value by Procurement Category ..............................137 Table 4.5. Average Elapsed Time: Issue of Specific Procurement Notice to Contract Signature ........138 Table 4.6. Africa: Numbers of Iterations and Average Response Times per Procurement Transaction Request, 2009–12 ...............................................................................................................................141 Table 4.7. Africa: Response Times for Procurement Categories by Clearance Level, 2009–12..........142 Figures Figure 1.1. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators .................15 Figure 1.2. Bank UCS Pilot Participants – Approaching the World Bank System ..................................18 Figure 2.1. CDD and Other Projects—Performance in Low Governance Environments .......................45 Figure 2.2. Procurement Performance and Development Outcomes by Type of CDD ..........................46 Figure 2.3. Trends in IBRD/IDA and RETF Support to FCS (2003–12) .................................................51 Figure 2.4. World Bank Group: ICT Portfolio 2003–10 ..........................................................................60 Figure 2.5. World Bank PPP Projects—July 2002 to December 2011, by FY........................................74 Figure 3.1. Country and Project-Level Prior Review Thresholds by Region—Civil Works and Goods ..97 Figure 3.2. Procurement Risk Ratings for Projects Above and Below Country-Level Prior Review Thresholds .............................................................................................................................................97 Figure 3.3. Distribution of Risk Ratings in Post-Reviewed Contracts...................................................100 Figure 3.4. Comparison of Project Procurement Risk (from P-RAMS) and PPR-Based Risk Ratings .101 Figure 3.5. Procurement Risk Categories, Risk Mitigation, and Risk by Sector—A Sample ................106 Figure 3.6. P-RAMS Completed by Month (2009–12) .........................................................................109 Figure 3.7. Summary Results from P-RAMS Applications, 2010–12 ...................................................112 Figure 3.8. P-RAMS Risk and P-RAMS Risk Mitigation.......................................................................112 Figure 3.9. Time Distribution of Complaints Resolution .......................................................................115 Figure 3.10. INT Complaints by Stages in Procurement and Project Implementation .........................117 Figure 3.11. Correlation of INT Cases with Project, Procurement, and Financial Management Risk (cases with at least one risk rating)......................................................................................................119 Figure 4.1. Number of Observations by Country, Fiscal Year, Procurement Category, and Sector.....136 Figure 4.2. IBRD and IDA Countries—Elapsed Procurement Times by Procurement Method (days) .139 Figure 4.3. Distribution of Days from Issue of Specific Procurement Notice to Contract Signature .....139 Figure 4.4. Borrower Delays and Bank Response, 2010–12 ...............................................................143 vi Abbreviations ADB Asian Development Bank MAPS Methodology for Assessing AfDB African Development Bank Procurement Systems CAS Country Assistance Strategy MCC Millennium Challenge Corporation CDD community-driven development MDB multilateral development bank CPAR Country Procurement Assessment NCB national competitive bidding Report NTLA non-lending technical assistance CPIA County Policy and Institutional OECD Organisation for Economic Co- Assessment operation and Development- FCS fragile and conflict-affected situation Development FICID International Federation of Consulting OECD-DAC Development Assistance Engineers Committee, Organisation for IAD Internal Audit Vice Presidency Economic Co-operation and ICB international competitive bidding Development process OPRC Operational Procurement Review ICT information and communications Committee technology OPSOR Operations Risk Management IDA International Development Agency Department, Operations Policy and IDB Inter-American Development Bank Country Services Vice Presidency IEG Independent Evaluation Group ORAF Operational Risk Assessment IFC International Financial Corporation Framework IFI international financial institution PAD Project appraisal document IIFCL India Infrastructure Finance Company PPP public-private partnership Limited P-RAMS Procurement risk assessment IGR Institutional Governance Review management system INT Integrity Vice Presidency RPM Regional procurement managers IPR Independent Procurement Review SEPA Procurement plan execution system ISN Interim Strategy Note UNCITRAL United Nations Commission on ISR Implementation Status Report International Trade Law WBI World Bank Institute vii Acknowledgments This report of the Independent Evaluation Group (IEG) was prepared by a core team led by Anjali Kumar, with major contributions provided by Nils Fostvedt, Prem Garg, Gita Gopal, Ian Hume, Eduardo Maldonado, Arvind Nair, Carla Pazce, Juliane Piecha, Robert Rothery, Swizen Rubbani, Justin Sigman, Kathryn Steingraber, Nadine Tushe, Christine Wallich, and Clay Wescott. Eleven country case studies were undertaken by Gian Casartelli, Jorge Claro, Ajay Guha, Rodolfo Hernandez, Jean- Jacques Lecat, Robert Rothery, Paul Schapper, and Christine Wallich. Additional contributions were provided by Jeanmarie Fath Meyer and S. Janakiram. Corky de Asis, Aimée Niane, Agnes Santos, and Lily Tsang provided administrative support. The team gratefully acknowledges the support of managers and staff throughout the World Bank, especially in Operations Policy and Country Services departments and from both former and current regional procurement managers. IEG appreciates the many individual procurement staff throughout the Bank who have been generous with their time for surveys and interviews on aspects of the Bank’s procurement system. Particular thanks are offered to the field-based procurement staff that provided assistance for field visits in Azerbaijan, Bangladesh, Ethiopia, Indonesia, Mexico, Morocco, the Philippines, Peru, Senegal, Tanzania, and Turkey, arranging meetings with key officials as well as private persons. IEG also extends its thanks to other areas of the Bank consulted during the course of this evaluation: the Controllers unit, Economic Management, Institutional Integrity, Internal Audit, and Legal units. IEG also appreciates exchanges with the African Development Bank, the Asian Development Bank, the European Bank for Reconstruction and Development, the European Commission, the European Investment Bank, the Inter-American Development Bank, and the Organisation for Economic Co-operation and Development, who shared their perspectives with the evaluation team, and its exchanges with academia at George Washington University, as well as Harvard University, the University of Florida and Nottingham University, UK. The team also thanks client government counterparts, private sector suppliers, project implementation units, personnel from implementing agencies, and members of civil society who participated in IEG’s surveys and interviews. The evaluation benefitted from constructive advice from external advisers and peer reviewers. Peer reviewers included Praful Patel (former Vice President, South Asia Region, and former Director, Infrastructure, Africa Region); Christiaan Poortman (former Vice President, Middle East and North Africa Region, World Bank, and current Senior Advisor at Transparency International and Chair of the Board of the ix ACKNOWLEDGEMENTS Construction Sector Transparency Initiative); and Dr. Peter Trepte (Senior Fellow in Public Procurement Law, University of Nottingham). An external advisor provided overall guidance to the study—Professor Daniel Gordon, Associate Dean for Government Procurement Law, at the George Washington University Law School. The evaluation was conducted under the guidance of Nicholas York, Director, and Ali Khadr, Senior Manager, IEG Country, Corporate, and Global Evaluation, and Caroline Heider, IEG Director General. x 1. Bank Procurement and Development Impact Introduction to Part II—Focus and Rationale As described in the first part of this two-part evaluation of the World Bank and public procurement, conducted by the Independent Evaluation Group (IEG), the World Bank has a dual role with regard to public procurement in its client countries. First, because good national public procurement practices are a major determinant of the effectiveness of public expenditure, building client countries’ capacity for better public procurement within their own national systems is an essential element of the poverty reduction focus of the Bank, and in accordance with global principles of aid effectiveness. The Bank’s contributions to country procurement capacity building are therefore the focus of Volume I of this report. Second, World Bank–financed investment lending operations constitute a significant element of public procurement in client countries; therefore, the Bank’s own procurement policies and processes affect the development impact of Bank lending. This volume therefore focuses on the second aspect of the Bank’s dual role: the extent to which the Bank’s own procurement guidelines and processes help achieve the Bank’s development goals, that is, to ensure that its funds are used effectively and efficiently and for the purpose intended. IEG therefore evaluates the extent to which the guiding principles of the present Bank Guidelines are achieved, in terms of transparency, competition, economy, and efficiency, and the development of domestic markets. The evaluation parallels an intensive multiphase review by management of the Bank’s procurement function, intended to pave the way toward significant future changes (Box 1.1). IEG’s evaluation is intended to inform management’s review in terms of specific proposals for change; also seeks to address specific queries and concerns raised by the Bank’s Board members regarding the Bank’s procurement systems and processes. Accordingly, this chapter first selectively reviews areas where country client and staff stakeholders have drawn attention to the need for review of the current guidelines and processes. Second, it examines evidence on core themes proposed by Bank management for inclusion in its future procurement framework (the use of country systems, the potential for engagement in upstream and downstream aspects of procurement, and, centrally, the overarching value for money proposition). Third, IEG selectively reviews aspects of Bank procurement policy of interest to the Board, for example, the scope for sustainability or environmental considerations or the scope 1 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT for domestic preferences. The balance of the report addresses the core themes of the extent to which current guidelines and processes meet the new business needs of the Bank (Chapter 2), the management of fiduciary risk in procurement (Chapter 3), and the efficiency of the current procurement system (Chapter 4). Box 1.1. Management Review of Bank Procurement Systems Management’s review of the Bank’s procurement systems began with a detailed memorandum discussed with the Bank’s Board in February 2012 that pointed toward changes in the relevance of the present system, away from traditional infrastructure projects for which its procurement systems were designed. These trends have paralleled global changes in client country capacity, supplier patterns, and new practice emerging in public procurement. The goal of the review, as stated by management, is to position the Bank’s procurement policies and procedures in the context of its modernization agenda, moving away from “one-size-fits-all” and at the same time reflecting its leadership role in promoting best practices in public procurement. Management points out that there has not been a fundamental review of the Bank’s procurement policies, starting from first principles, since the Bank’s founding. In its most recent discussions with the Board (April 2013), the Bank presented an overall vision for its proposed future reforms: Procurement in Bank operations supports clients to achieve value for money in delivering results. Developmental results achieved are core, and the overarching goal of achieving value for money is underpinned by specific areas of change: an enhanced but selective use of country systems at the request of the borrower, under a new best-fit-for-purpose approach; a changed, nonmandatory role for the Bank Guidelines and standard bidding documents; introduction of new procurement methods; a greater strategic procurement planning function; and the maintenance of fiduciary assurances despite a smaller number of prior-reviewed contracts, through a more rigorous risk- based approach. At borrowers’ request, particular methods and procedures of the procurement systems of the implementing institution of a specific operation could be selectively used under the new fit for purpose approach. Sources: World Bank 2012c, 2013b. Main Findings  On the whole, country clients, the private sector, and Bank staff agree that the Bank’s present procurement guidelines for goods and works are reasonably successful in securing fairness, competition, and transparency in Bank procurement. Nevertheless, there may be scope for improvement in certain details, in addition to 2 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT areas of complex procurement. There are greater perceptions of problems with the consulting guidelines.  There is considerably less comfort with Bank procurement processes, especially with regard to time taken, flexibility, and consistency, which lead to losses in development effectiveness. Focus on compliance in transactions may distract from a focus on outcomes and need not ensure the containment of systemic fraud and corruption.  Future options for the use of country systems in Bank procurement can draw lessons from its recent pilot, an exercise characterized by rigor but also by minutiae in terms of detailed of areas of compliance.  Some areas of difference between the Bank and its clients touch on core principles (competition, market access); others impact such principles less. A review of the materiality of such differences would inform the adoption of an incremental approach toward use of country systems, in line with other multilateral development banks (MDBs).  Country stakeholders and the private sector do not invariably prefer the use of country systems, especially for the kind of large value contracts for which Bank systems were designed.  Regarding upstream involvement, there is scope for more Bank support in acquisitions planning and in the integration of procurement in public budget management.  There is a clearly perceived need for greater involvement downstream in terms of contract management. The roles of different players need review relative to international practice, legal obligations, and task team leader incentives.  Most countries have elements of domestic preference in their procurement policies. Although the Bank Guidelines endorse the principle of developing domestic manufacturing and contracting capabilities, the prescribed domestic preference scheme is restrictive in scope, which may explain its limited use. To the extent that the Bank retains the objective to develop domestic capacity through its procurement system, it may wish to consider a broader approach.  With regard to sustainable and environmentally aware procurement, Bank Guidelines and practices already offer many avenues to incorporate “green” considerations, although they are not systematically used. There is scope for more 3 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT explicit incorporation of environmental factors in technical specifications and bid evaluation criteria.  The principle of value for money in procurement is being increasingly incorporated into public procurement worldwide, with varying emphasis. Many aspects are implicit in Bank procurement; some are frequently used. Stronger direction could be given to staff and borrowers to fortify the adoption of such practices. Bank Procurement Guidelines: Overall Perceptions and Specific Concerns By and large, both country clients and other users of the Bank’s procurement guidelines view them positively (Box 1.2). Procurement under Bank Guidelines is seen to produce good results in terms of achieving open and fair competition, integrity, transparency, quality, and price—often better than national systems operating within the context of national oversight and accountability (Appendix Tables A.1–A.6). Some country clients perceive that serious bidders are more likely to show interest if a project is Bank financed and under Bank procurement policy, as opposed to national procedures. The Bank was deemed to balance different considerations and not limit decisions to price factors—although these factors play a significant role. Box 1.2. Procurement—Overall Perceptions of Bank Guidelines and Processes In Morocco, private sector representatives have confidence in the Bank’s procurement system for reasons ranging from safe financing and payment to transparency and integrity to possibility of recourse. In Azerbaijan and Tanzania, Bank Guidelines were thought to bring better quality by eliminating unqualified bidders and because payments under Bank-supervised projects were generally faster. Discussions also suggested that bidders appear to take fiduciary concerns more seriously in internationally funded procurement. There was general consensus that Bank Guidelines work well for straightforward procurement, especially in infrastructure projects. In Azerbaijan, for example, project implementing units in charge of infrastructure projects cited the guidelines positively in terms of overall content and levels of participation and also voiced appreciation for the quality of standard bidding documents. By contrast, project implementing units in health and education projects were critical of the Bank’s rigid approach and its insistence on international competitive bidding process for goods and works, even for small items or projects, when clients deemed national competitive bidding to be equally effective. Source: IEG field visits. Yet despite broad acceptance, respondents to the IEG surveys and field visits pointed to specific areas of Bank Guidelines that may merit re-examination or revision. These 4 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT areas are discussed below. Less positive messages were conveyed in terms of Bank processes: country clients felt that there was at best modest flexibility to respond to special circumstances, and there were suggestions of delays in contract award. Bid Submission—The Two-Envelope System The two-envelope system used by entities such as the Asian Development Bank (ADB) is not currently a World Bank process. It refers to the simultaneous submission of technical and price proposals in separate envelopes, with sequential opening and evaluation. The Bank requires single-envelope bid submission (except for consultants), perhaps because of concerns about the fraud and corruption or perhaps because of possible lack of transparency. Procurement staff point out that the single- envelope system acts as a deterrent for high-quality firms, encouraging too many bids with a least cost focus. This perception is shared by some other donors, who also believe that the one-envelope system leads to outcomes where quality is largely ignored (evaluators are reluctant to reject technically inferior, lowest priced bids once prices are known). The ADB permits the two-envelope system. In Bangladesh and the Philippines, country procurement processes include single- stage two-envelope procurement. In Ethiopia and Tanzania, governments also preferred a two-envelope, postqualification system, as described in the preceding paragraph and used by other entities, rather than the Bank’s lengthy prequalification system with one envelope. Governments perceive the latter as very time consuming and limited in terms of screening out poor quality bidders/bids. They view the one- envelope system as militating against value for money. In Ethiopia, there was also substantial discussion about the Bank’s policy of requiring prequalification of bidders (rather than postqualification) and one-envelope bidding—an approach that makes it very difficult to turn down a low-price bidder that may not have the capacity to perform or that may be known to have performed poorly on a previous contract. 1 One option for the Bank could be to review existing bid management procedures and permit the two-envelope system, if deemed adequate. In terms of bid criteria, some countries request permission to use bills-of-quantities (that is, with estimates of quantities of materials, parts and labor to be used and their respective prices—also referred to as price-percentage bidding), which may be useful in a less-sophisticated environment. 2 Bid Evaluation—Third-Party Participation In Mexico, national legislation permits the use of “social witnesses” to participate, on a nonvoting basis, in bid evaluation; this practice is considered important for raising transparency. Respondents in Bangladesh point out that third-party participation has 5 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT been selectively used there, as in the Philippines. The Bank could consider how to make this selectively available, for example, through technical experts, to the extent that confidentiality considerations can be protected. 3 Some country clients have also requested that the Bank consider participation in bid evaluation, as a silent or nonvoting observer, to help ensure the integrity of the process. Although potentially value adding, this method nevertheless raises questions of possibly conflicting roles for the Bank, in terms of provision of advice versus provision of oversight. 4 New Procurement Methods—Framework Contracts, Using Negotiation There are many variants of framework contracts, and all allow the multiple use of suppliers, without rebidding each purchase. They can save cost and time for small but frequent purchases or for repeat or related purchases. The Bank’s current policy, which introduced the use of framework contracts in 2011, limits their use to common- use goods, simple nonconsulting services, or small-value emergency works, with a contract duration of up to three years and a value not exceeding the national competitive bidding (NCB) limit. Several countries confirmed that framework contracts are permissible to some degree under local law (Bangladesh, Mexico, Perus, and Tanzania), although their use has been limited under the Bank’s Guidelines. Several country respondents felt that there should be more flexibility to negotiate with a bidder, selected through a competitive process, on contracts for goods and works, as is the case with certain consultant selection methods (quality-based selection, for example). Country respondents said that this would increase clarity and provide an opportunity to tailor and correct minor omissions. Postbid negotiation was permitted in some countries, for example, in Bangladesh. Such practice makes it difficult to ensure transparency and is therefore not widely accepted. However, other options to introduce a degree of dialog into the procurement process, such as the competitive negotiation method offered in specific circumstances by the United Nations Commission on International Trade Law (UNCITRAL), may be considered. 5 The Bank’s position on negotiation options was considered restrictive (see Chapter 2). In Azerbaijan, Bank bidding processes were perceived as overly emphasizing the need to keep bidders and buyers at arm’s length from each other, allowing dialog only through formal meetings. Direct dialogue between buyer and bidder was thought to be not sufficiently encouraged, thereby restricting the exchange of important information that could improve bids. Decisions related to the degree and methods of negotiation could be made on a case-by-case basis in a country-specific context. 6 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Cofinancing Several borrowers complained about the imposition of Bank Guidelines even when the Bank finances only a small share of a project. In Morocco, in a project cofinanced by the African Development Bank (AfDB), the AfDB Board had to be requested to allow nationals of nonmember states to participate in the entire procurement, although the Bank was a minority contributor. In another case, a sector-wide approach, the Bank’s requirement to use its own procurement guidelines was rejected by the government (especially with regard to allowing the Bank to audit suppliers’ accounts), leading to protracted delays and eventual project restructuring. Consultant Selection Although the Bank Guidelines for procurement are generally seen as reasonable and value adding, its consultant’s guidelines appear to be less accepted. Many concerns surfaced during country visits: the lengthy quality and cost-based selection process and the volume of information sought from bidders, which were seen as expensive to produce and time consuming to evaluate. It was felt that better consultants refrained from expressing interest in quality- and cost-based selection processes. There were also concerns about the outcomes, which were perceived to lead to the selection of the “least worse” firm, thereby affecting the quality (that is, value) of services. Clients pointed toward undue attention to and incorrect use of proposal prices, apparently reflecting Bank insistence on quality- and cost-based selection rather than allowing greater use of quality-based selection. Other difficulties have been mentioned, for example, geographical diversification in short-listing criteria in situations where only local firms were likely to bid. In a quality- and cost-based selection process, the Bank correctly discourages any negotiations of the scope/quality of services and of prices, because the quality/price of the proposals is compared in the bids. Yet in essence this problem arises because quality- and cost-based selection is quite often adopted when it should not be, that is, when services are complex and contract finalization requires discussion between client and bidder. It has also been pointed out that the post- contract-award negotiation system in consultant contracts at the Bank, following a quality and cost-based selection, is not widely accepted as good practice. Greater use of a quality-only system was urged. 6 Negotiations are allowed under quality-based selection to define contract scope and contract price. 7 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Bank Procurement Processes Delays in Prior Review and “No Objections” Although the Bank Guidelines are generally seen as reasonable and value adding, the Bank’s procurement processes are perceived to be problematic and time consuming. Delays in getting no objections in the prior review process was a familiar theme in almost all country visits. Overall, the Bank was seen as placing more emphasis on safeguarding against risks to the integrity of the process than on efficiency and time. Countries expressed concern about the Bank’s lengthy process of approvals, especially when reference is made to the regional procurement manager or to the Operational Procurement Review Committee (OPRC) (Appendix Table A.6). Conflicting Guidance In several country visits, clients raised issues regarding inconsistencies in the advice and decisions among different Bank staff. Respondents in Mexico commented on the inconsistencies in advice from procurement personnel and task team leaders, compounded by frequent rotation among the latter. Consistency in comments was a significant issue for an implementing agency in Tanzania. Document “churn” was also a concern. Even when response times meet business norms, the Bank sometimes requests more documentation or clarification, extending clearing time. 7 Written communication requirements particularly affected decentralized projects (for example, in Indonesia). Implementing agencies commented that delays are more common on Bank-financed contracts than in those financed by ADB, AfDB, or the Inter-American Development Bank. Inflexible Interpretation of Guidelines Although the Bank Guidelines were deemed to work reasonably well and to incorporate sufficient provision for flexibility, it was generally felt that they were inflexibly applied (see Box 1.3), with the Bank not able to respond quickly to special circumstances (Appendix Table A.5). Clients urge focus on underlying intent and the application of reasonable interpretation. Finally, many stakeholders—Ethiopia, Indonesia, Morocco, and Tanzania were notable examples—believe that focus on transactions displaces focus on achieving outcomes. It is pointed out that even with perfect compliance, there can be fraud and corruption, which is sometimes endemic. Compliance may sometimes give a false sense of security and may even obscure malpractice. It was urged that efforts be made to tackle fraud and corruption issues at a systemic level rather than through individual Bank projects. 8 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Box 1.3. Procurement Processes—Examples of Inflexibility and Loss of Development Effectiveness In Indonesia, the implementing agency involved in a power sector procurement, the State Electricity Company, did not advertise on the United Nations Development Business, as required by the Bank. The electricity company, an experienced entity, had advertised on its own website. Suppliers were aware of the tender, and competition was reasonable. In any event UNDB is little used by potential suppliers. However, the false step resulted in a one- year delay. The Bank considered declaring misprocurement but subsequently allowed the contract to go ahead. Task team leaders and country management concurred that it was obvious that the Bank should have waived the oversight, quickly, and proceeded with procurement. Two examples were offered in Mexico. The first referred to a failed loan for the supply of vaccines during an influenza epidemic. Contract review by the procurement Board at headquarters pointed out that required anticorruption clauses were not included in the invitation to submit bids. Authorities pointed to the urgency of the situation and requested flexibility from the Bank. And in Mexico’s energy efficiency lightbulb exchange program, proposed procurement solutions that could have made the program sustainable over time were rejected by the Bank because they required contracting with specific established businesses, deemed to be a noncompetitive process. In Tanzania the U.S. Agency for International Development had contracted with a social marketing firm to market a premixed nutrition formula in different parts of the country. The firm was seasoned and had developed an effective campaign. According to the country client, Bank financing was unable to piggyback on the existing contract, although rebidding would have implied that two parallel social marketing contracts would be in place, with duplication of effort, training, materials development, and corporate overhead. However, according to the task team leader, single-source procurement would not have been a sensible option for this consultancy, as it was not a continuation of existing work but rather different approaches to the same intervention in different parts of the country. Examples also illustrated inflexibility in terms of bid securities and bid securing declarations. To ensure that bidders honor their bids if they win, the guidelines offer borrowers the option of requiring bank guarantees (with the amount and form to be specified in the bidding documents). However, the guidelines also give borrowers the option of requiring a bid securing declaration. Instances arose (Morocco) where Bank staff have been demanding in terms of the wording of the guarantee, leading to the disqualification of bidders. And in Tanzania, suppliers complained that Bank procurement staff insisted on bid security, instead of the bid securing declaration. In Indonesia, a bid was rejected because the bid security validity was two days less than prescribed, which Bank staff considered a de minimis deviation. Source: IEG field visits. 9 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT To summarize, although overall perceptions of Bank Guidelines are fairly positive, there is often discomfort with the interpretation of the guidelines and with the procurement process, which is seen as cumbersome, time consuming, prone to delay, and inflexible. Using Country Procurement Systems Given the pros and cons of current Bank procurement rules and processes, to what extent would it be advisable for the Bank to move toward greater use of country systems, and if so, in what manner should it proceed? In view of the centrality of this topic to the ongoing debate on future directions for the Bank, IEG presents a detailed analysis, beginning with a review of the Bank’s use of country systems pilot and analysis of its outcomes, followed by an examination of practices in other bilaterals and international financial institutions (IFIs), and finally, reviewing perceptions presented by the Bank’s stakeholders, specifically on the Bank pilot as well as on the general proposal to move toward country systems, based on evidence gathered from field visits. THE BANK’S USE OF COUNTRY SYSTEMS PILOT Beginning in 2005, the Bank sought to further increase its reliance on country systems, in particular for the remaining frontier of international competitive bidding (ICB) (World Bank 2005b). At the request of the Board, Bank management prepared a methodology on how to identify countries and projects to participate in a limited piloting program. It cited the core rationale for the use of country systems in its presentation to the Board in June 2007: maximizing development impact, increasing country ownership, 8 facilitating harmonization, and reducing transaction costs. 9 The Board approved the methodology in April 2008, and formal implementation of the program began in FY09 (World Bank 2009c). Twenty countries expressed initial interest; 17 remained engaged at the close of the pilot program. Over a period of two years, the program would identify a group of 8–10 countries and a set of projects within those countries where national procurement systems were of sufficient quality and capacity to achieve results comparable to the World Bank’s procedures. The first year progress report (World Bank 2009c) suggested delays. Early assessments revealed that all participating countries needed some further development of their procurement systems to meet Bank standards. Management extended the program to June 2011 and proposed that it introduce country-level capacity development and interim mitigation measures for projects to address deficiencies, allowing countries conditional approval for the pilot, subject to compliance with a Country Development Action Plan. 10 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT A core element of the methodology was the use of the new instrument of the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD–DAC), introduced for the benchmarking of procurement systems: the Methodology for the Assessment of National Procurement Systems (MAPS)—devised with considerable Bank input (see Volume I, Chapter 3) (OECD 2005b, 2006b). It offered a platform for the benchmarking of countries’ procurement systems and was therefore used by the Bank for the establishment of baselines for its analyses of the extent of countries’ readiness to use their own procurement systems in World Bank-financed projects. 10 The piloting program methodology had three stages: Stage I: Selection of potential pilot countries and assessment of the overall quality of each country’s procurement system through a MAPS exercise Stage II: Assessment of the consistency and equivalence of the country’s procurement system, compared with the Bank’s procurement policies Stage III: Assessment of compliance, performance, capacity, and fiduciary risks at the sector/executing agency level. Fifteen countries completed the Stage I evaluation of country procurement systems using MAPS. The Bank drew up precise guidance on the passing scores required to meet Bank standards for each subindicator. Thus, 17 core subindicators bearing on key transparency and economy principles of the Bank and procurement policy were required to receive the highest score of 3. Another 10 required a score of 2, subject to an action plan to achieve a 3 during implementation. An additional 25 subindicators could earn a score of 2 without corrective measures. There were also two subindicators where only a score of 3 or 0 was possible, regarding the regulatory body and complaints review body. 11 IEG calculates that this implied a passing score of a total of 128 out of a possible 162 points. 12 Stage II, completed by nine countries, analyzed whether candidate countries’ procurement policies and procedures were consistent and equivalent to those of the Bank. The assessment was independent of Stage I, notwithstanding similar sources of information. One element of Stage II included a further stage of analysis, to see whether pilot countries’ bidding documents could be used for international competitive procurement. 13 A checklist of clauses consisted of 44 subcategories for instructions to bidders and an additional 37 for general conditions of contract. 14 Stage III entailed an assessment of compliance, performance, capacity, and fiduciary risks of the executing agencies of proposed Bank pilot projects. No countries completed Stage III of the Bank’s use of country systems pilot. 11 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT No countries were unconditionally cleared to participate at the close of the piloting program. The second progress report in December 2010 admitted that the piloting program had proved challenging and had “not been a success” from the perspective of its original objective—use of country procurement systems in individual projects (World Bank 2010e). It was evident that no project using country procurement systems would be approved before the scheduled end of the program on June 30, 2011, and the Bank decided that the program would not be extended beyond that date. Of the four countries conditionally cleared to move to the project phase, only one, Brazil, was moving forward with Phase III at the time of management’s final progress report. African regional and country management teams elected not to proceed with project identification in Mauritius, Rwanda, and Senegal because of the uncertainty of the future of the program, and also so they could focus on the proposed new Program for Results agenda (Table 1.1). Table 1.1. Summary of Results of the Bank’s Use of Country Systems Pilot (2005–10) Particip- Participants Stage achieved ants (nos.) (countries) Countries cleared to proceed to project phase subject to CDAPs 4 Brazil, Mauritius, Rwanda, and Senegal Countries/Agencies that completed Stages I and II and were 3 Morocco, India Powergrid, the recommended to continue to pursue procurement reforms with support Philippines from the Bank Other Countries completing Stages I and II 2 Macedonia, Poland Countries that had only completed Stage I at the end of the UCS Pilota 6 Burkino Faso, Colombia, Ghana, Indonesia, Jordan, and Panama Countries pursuing other prioritiesb 2 Bhutan, Turkey Total 17 Source: IEG analysis of World Bank data on use of country systems pilot. Note: CDAP = country development action plan; UCS = use of country systems. a. Though Indonesia is included as a participant in Bank documents, its situation seems ambiguous. Indonesia conducted a MAPS exercise, though it was not formally a part of the UCS pilot. b. Bhutan’s small size and nature of engagement with donors prompted it to first focus on the harmonization of donor and country policies, rather than UCS; Turkey, as an European Union accession candidate country, opted to give greater emphasis to the harmonization of its procedures with those of the European Union, with support from the European Union under its SIGMA program. The second progress report acknowledged certain factors that contributed to the limited outcome of the pilot program, notably, “the uniqueness and specificity” of some Bank standards that were a particular obstacle to Stage II; high costs and uncertain benefits for participating countries and Bank staff; skills needs on system assessments; and in some countries, the inapplicability of the concept of a country system, especially in large federal countries, in the presence of variations between states, sectors, and implementing agencies. Yet the exercise had been a learning process. The methodology allowed the Bank to bring together baseline assessments 12 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT of multiple countries’ procurement systems in a single, comprehensive framework and exposed ways that diagnostic instruments, MAPS in particular, could be improved. The process of dialog with participating countries, other MDBs, and within different Bank departments was also valuable. IEG ANALYSIS OF THE BANK’S USE OF COUNTRY SYSTEMS PILOT IEG’s analysis of the Bank’s use of country systems piloting program aims to (i) review the quality of participating countries’ procurement systems, to understand to what extent Bank expectations could not be met; (ii) review Bank criteria for countries’ compliance under the piloting program, to assess how easy or difficult it was to satisfy those standards; and (iii) examine the extent to which the Bank could have provided more flexibility in these requirements in order to enable moving forward with the use of country systems experiment. This was undertaken by a detailed analysis of all diagnostic material emerging from the experiment, for each of the two principal stages—the Stage I and Stage II assessments—and a review of Bank Country Procurement Assessment Reports (CPARs) and loan documents in sample countries, to examine areas in which countries’ national procurement systems have not met Bank norms. Stage I MAPS Assessments—Review of Use of Country Systems Documents The Stage I analysis focused on scores of each country for the 54 subindicators of MAPS, in comparison to the minimum passing scores designated by the Bank’s piloting program (Appendix A, section on MAPS indicators and scores). Of the 16 countries that produced reports for the Stage I analysis, IEG calculated the average country scores for each indicator and the number of countries receiving passing scores. 15 Pillar I, the Legislative and Regulatory Framework, was both the strongest performing in absolute terms (with an average score of 2.4 across all subindicators for all countries) but also the weakest in terms of numbers of countries receiving passing scores—around 11 countries. This reflects the fact that Pillar I contained a disproportionate number of “core” subindicators requiring the highest score of 3 in order to advance in the piloting program.16 The second and third major pillars, Institutional Framework and Management Capacity and Procurement Operations and Market Practices, had average scores at the required average of 2.2, but only about 12 countries met this target—although some exceeded it. One area where there were significant discrepancies in Pillar III was with regard to contract administration and dispute resolution provisions. The Bank’s average passing score, at 2.7, exceeded the country average of 2.2, and only 9 13 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT countries passed on this score. Under Pillar IV, Integrity and Transparency of the Public Procurement System, the efficiency of access to appeals mechanisms was a difficult area, with a required average passing score of 2.8, and an achieved average of 2.1 (Table 1.2). Table 1.2. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators Average number of Average Average countries with score passing score passing score (out Pillar and indicator received required of 16) Pillar I: Legislative and Regulatory Framework 2.4 2.6 11.1 Indicator 1: Public procurement legislative and regulatory framework 2.5 2.9 10.5 achieves the agreed standards and complies with applicable obligations. Indicator 2: Existence of Implementing Regulations and 2.3 2.2 11.8 Documentation. Pillar II: Institutional Framework and Management Capacity 2.2 2.2 12.1 Indicator 3: The public procurement system is mainstreamed and 2.2 2.0 13.3 well integrated into the public sector governance system. Indicator 4: The country has a functional normative/regulatory body. 2.5 2.5 13.3 Indicator 5: Existence of institutional development capacity. 1.8 2.0 10.0 Pillar III: Procurement Operations and Market Practices 2.2 2.2 11.3 Indicator 6: The country's procurement operations and practices are 2.0 2.0 12.3 efficient. Indicator 7: Functionality of the public procurement market 2.3 2.0 11.5 Indicator 8: Existence of contract administration and dispute 2.2 2.7 9.7 resolution provisions Pillar IV: Integrity and Transparency of the Public Procurement 2.2 2.4 11.3 System Indicator 9: The country has effective control and audit systems. 2.0 2.0 12.4 Indicator 10: Efficiency of appeals mechanism. 2.1 2.8 9.6 Indicator 11: Degree of access to information 2.4 2.0 14.0 Indicator 12: The country has ethics and anticorruption measures in 2.4 2.6 11.4 place. Sources: World Bank UCS country documents. Performance of individual countries was mixed, ranging from scores of 1.4 for Bhutan to 2.7 for Turkey and Morocco (Figure 1.1). 17 None met all required benchmarks for all subindicators, yet half (Brazil, Macedonia, Mauritius, Morocco, the Philippines, Rwanda, Senegal, and Turkey) had total scores greater than the total required score. For the three weakest subindicators, fewer than half of the countries met the required benchmark. 18 Notable difficulties were encountered in the area of nondiscriminatory participation and selection, where 5 of 16 countries received a passing score. 19 Issues clustered around two areas: state-owned enterprises and regional/domestic preferences, especially the former. In contrast to the MAPS standard of requiring rules for participation of state-owned enterprises to promote 14 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT fair competition, four countries (Ghana, Poland, Rwanda, and Senegal) allowed unrestricted participation, while the rules in a fifth (Turkey) were overly lenient. 20 Figure 1.1. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators Sources: IEG analysis of World Bank UCS pilot country documents. Price preferences given to domestic supplies were another area of contention. The MAPS benchmark on domestic price preferences limited such preferential treatment to a reasonable amount (for example, 15 percent or less). A number of countries (Burkina Faso, the Philippines, Senegal, and Turkey) allowed higher levels of preferential treatment. Price preferences for domestic firms, minimum locally manufactured content, or required association or purchasing from domestic firms also served in some cases to lower scores. Another area of common failing was subindicator 1(h) on complaints arising from the failure to have a truly independent review body (Indonesia, Morocco, and the Philippines). Only 7 of 16 countries received a passing mark. IEG undertook a further analysis of the extent to which such issues are encountered in other Bank borrower countries, beyond those participating in the use of country systems pilot. IEG’s review covered all additional selected sample countries for this evaluation (Appendix Table A.8). Given that the Bank has sought to expand the use of country systems through the use of NCB, IEG reviewed recent projects documents as well as CPARs, in terms of their provisions for procurement under the NCB method. 21 According to Bank Guidelines, country procurement rules can be used under NCB as long as they are broadly consistent with the Bank’s principles. 22 In specific areas where local laws and practices do not comply, the Bank details measures to be taken to enable country procurement rules to be used. These modifications are typically included in the loan/credit agreements, and where applicable, in bid documents. 15 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT A review of such additional requirements by the Bank for the use of NCB revealed a list of nearly 40 problem areas (Appendix Table A.8). 23 Many were common to those that emerged from the pilot, although with somewhat more granularity and specificity. The most common areas requiring modification included the Bank’s prohibition on regional and domestic preferences, eligibility restrictions (including, for example, the need for foreign firms to enter into joint ventures with local firms), advertizing and free availability of bidding documents, time permitted for bid preparation, 24 the use of appropriate bidding documents, 25 and the award of contract to the lowest priced responsive bidder 26 without further negotiations. 27 Overall, the analysis confirmed that there are a number of areas in which many Bank client countries follow practices that are different from those of the Bank. Some of these may be more important than others, such as the principle of non-discrimination and the level of domestic preferences. Others may however be easier to accommodate, such as the Bank’s requirement for single envelope bid submission. Some derogations that allow greater use of negotiation are used elsewhere, though they are generally discouraged in the Bank. Stage II Equivalence Analysis—Review of Guidelines In Stage II, the piloting program sought to establish, in key areas of countries’ national procurement systems, equivalence to key features of the Bank Guidelines (Table 1.3). IEG prepared a mapping among key sections of the Bank Guidelines, corresponding policy requirements for the piloting program, and the evaluated performance of participating countries. Scores were assigned: 3 for areas where no problem was noted, 2 for areas where country policy deviated in a minor or easily remedied way, and 1 for a significant nonconformity. 28 Despite some overlap in subject matter (for example eligibility), the relative scores of countries varied from Stage I (Figure 1.2 and Box 1.5). A list of all areas assessed and their scores for conformity to the Bank Guidelines can be found in Appendix Table A.9. Table 1.3. Bank Requirements for NCB and Derogations in Select Countries No of reports No of countries reviewed reviewed where where the Issue the issue is Bank requirements for NCB is noted noted (35) (26) There may be no regional /domestic preferences regarding the sources of labor and 18 15 material. Prospective bidders must be allowed at least 30 days for bid preparation. 19 14 Appropriate standard bidding and prequalification documents must be used. 15 14 Eligibility cannot be restricted based on nationality of bidder and/or origin of goods. 17 13 Award must be made to the lowest evaluated qualified and responsive bidder. 15 13 16 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Minimum requirements must be explicitly stated in the documents. 12 12 Bidding opportunities must be advertised in the local press. 11 11 Bidders are not generally required to register with a local or federal authority except under 13 11 certain conditions. The procurement process cannot be cancelled, all bids rejected, and/or rebidding 13 11 conducted without approval. Public bid opening is required. 11 9 Bid evaluation criteria other than price may be allowed only if quantified in monetary terms. 8 8 Price negotiations may not be conducted with “winning” bidders prior to contract signature. 10 8 Parastatals may only be allowed to bid under certain conditions. 9 7 Bids may not be rejected based only on a comparison with the procuring entity's estimate; 11 7 invitations to bid shall not establish minimums and maximums. Audits and inspection of records related to bid submission and performance of the supplier 8 7 are permitted. “Two-envelope” bid opening procedure is permitted for procurement of goods or works 6 6 under specific conditions, notably if domestic law precludes use of one envelope. Foreign firms' eligibility cannot be conditioned on joint ventures with local firms. 7 5 Qualification criteria shall be applied on a pass/fail basis. 5 5 Extension of the time period to prepare bids may only be allowed under exceptional 5 5 circumstances. An inflation clause is recommended for contracts over a year. 7 5 Bidding documents are freely available. 7 4 Bid security shall be in the form of a letter of credit or bank guarantee from a reputable 5 4 bank. Joint venture partners must be jointly and severally liable. 6 3 There may be no restrictions on the means of delivery of bids. 3 3 Prequalification should be used only for large works projects. 5 3 Award must be published. 4 3 Bidding documents and contract shall include provisions on sanctions for fraud and 3 3 corruption. No preference may be given to suppliers or contractors based on region or locality of 4 2 registration, small size, ethnic ownership, and so forth. Source: IEG data compiled from loan agreements and CPARs. Note: NCB = national competitive bidding. A common problem area in Stage II, again, related to the restriction of participation of state-owned enterprises. Another problem area under eligibility was the lack of definition of conflicts of interest. Other issues that scored below 2.5 out of 3 included provisions on two-stage bidding and turnkey contracts; notification and advertisement; and examination, evaluation, and comparison of bids. 17 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Figure 1.2. Bank UCS Pilot Participants – Approaching the World Bank System Senegal Poland Bhutan Morocco Philippines Macedonia Colombia Rwanda Mauritius Brazil 2.4 2.5 2.6 2.7 2.8 2.9 3 IEG Scores Source: IEG analysis of World Bank UCS pilot country documents. Box 1.4. Stage II Equivalence Analysis – Bidding Documents Review Of the 11 countries for which IEG reviewed Stage II, 7 conducted a specialized bidding documents review. Four (Bhutan, Mauritius, Rwanda, and Senegal) had bidding documents modeled after World Bank standard bidding documents. Such documents for these countries were found to be acceptable (although with some exceptions and clarifications). Among the three others, one (Macedonia) had many problem areas; the other two (the Philippines and Poland) had limited issues that were comparable to countries with bidding documents modeled after the Bank’s. Bidding documents of all seven countries had some deviation from the Bank Guidelines/best practice, though many problem areas were not necessarily viewed as incompatible with the Bank’s requirements for the pilot program. Recurring issues included: • Differences in provisions on eligibility and sanctions. Firms sanctioned by the Bank were not necessarily excluded by the pilot countries (Mauritius, Poland, and Senegal). Descriptions of conflicts of interest in the case of state-owned enterprises (Macedonia, Poland, Rwanda, and Senegal) were also a problem area; sections on bidder’s nationality and related cross reference to eligible countries were frequently omitted. • Inadequate fraud and corruption provisions. There were no such provisions for Macedonia and Poland (although reports note that in Poland this is covered by the Polish Penal Code; in Macedonia, this is indirectly covered under other provisions), and they were lacking in detail in Rwanda. The definition of “obstructive practices” was different from the Bank standard in the case of Senegal and was missing in Rwanda. • Neglect to require statements of source of funds (Bhutan, Macedonia, and Poland). • Need for more comprehensive guidance concerning the use of bidding documents (for example, Macedonia and Rwanda). Source: IEG field visits. By the end, four pilot countries (Brazil, Mauritius, Rwanda, and Senegal) were cleared to proceed to the project phase subject to action plans (World Bank 2011f). Five (the India Powergrid, Macedonia, Morocco, the Philippines, and Poland) were not cleared after Stage I and Stage II assessments. Of these, the Bank suggested that Morocco, the Philippines, and the India Powergrid could continue to pursue 18 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT procurement reforms with Bank support. Six countries (Burkino Faso, Colombia, Ghana, Indonesia, Jordan, and Panama) had completed only Stage I at the close of the pilot program. And two (Bhutan and Turkey) completed the assessment process but chose alternative paths to meet their needs. In the end, even the four countries conditionally cleared to move to the project phase did not do so. Brazil was expected to move forward at the time of the final progress report but shifted course after deciding that the conditions imposed as a requirement for the use of country systems, which effectively created a hybrid, modified national procurement system, were too onerous. Meanwhile, because of a desire to focus on the proposed new Program for Results agenda and the uncertain future of the program, regional and country management teams elected not to proceed in Mauritius, Rwanda, and Senegal. USE OF COUNTRY SYSTEMS – THE EXPERIENCE OF OTHER DONORS The World Bank was not alone in the obstacles it encountered in the use of country systems experiment (Box 1.4). Retrospective looks at the movement toward this note the limited capacity of national procurement systems as a common theme (Pallas and Wood 2009; Ministry of Foreign Affairs of Denmark 2008) across donors. Institutional policies, fiduciary responsibilities, and concern for the speed of disbursement all contributed to the reluctance to rely on country systems despite the consensus of the Paris Declaration. 29 Interestingly, however, the OECD has noted that there is no clear relationship between the use of country systems and the quality of national procurement systems, suggesting that the greatest hurdle to greater use of country systems may be political instead of technical (OECD 2011c). The United States’ government’s posture, in the policy of its Millennium Challenge Corporation (MCC), is pragmatic: [T]he use of country systems for elements of compact implementation can deepen country ownership. However, MCC recognizes that the desirable goal of country ownership achieved through the use of country systems should not be pursued at the expense of program results or fiscal responsibility and accountability (Blue and Eriksson 2011). In this respect, MCC is further constrained by the pressure to finish projects and obtain measurable results within the five-year time frame of a compact. If host government inefficiency, lack of experience, or outright corruption weakens MCC’s ability to finish the job, MCC chooses to take greater control of implementation (Blue and Eriksson 2011). 19 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Box 1.5. Use of Country Systems—The Bank and the Larger Development Community Impetus toward increased use of country procurement systems started in the larger development community, to lighten the burden on recipient countries of navigating specific procurement rules of each donor along with their own system, ease cofinancing, and motivate countries to bring their systems up to an acceptable international standard, thus serving development goals by improving all government expenditures, not just those funded by donors. Several donors, including the World Bank, adopted policies to promote the use of country systems. Yet there was little consensus in terms of what use of country systems implied or how it would be implemented. The Organisation for Economic Co-operation and Development report on OECD: Percentage of Aid using progress in implementing the Paris Declaration Country Systems, 2005–10 reported that use of country procurement systems by donors overall increased from 40 percent in 2005 to 44 percent in 2010. For the World Bank, use of country systems rose somewhat more, from 42 percent to around 54– 55 percent. These statistics are, however, clouded by the Paris Declaration’s ambiguous and broad definition of the use of country systems.a Results obtained for the World Bank reflected, in part, its increased use of national competitive bidding (NCB) for those goods and services and in those markets that would be unlikely to attract foreign competition. However, under NCB, borrowers are required to make adjustments to be in compliance with the Bank’s guidelines (World Bank 2011e, 2007b). NCB as a proportion of procurement methods has risen over time as countries’ domestic supply capacities mature. Other forms of Bank procurement also make greater use of national procedures, such as direct contracting, the use of force accounts, and loans through financial intermediaries. The Bank’s new Program for Results instrument proposes to make intensive use of country systems, based on an agreed capacity building plan.b In addition, development policy lending by the Bank is effectively channeled entirely through national systems, and an increased share of development policy loans would be reflected in the above statistic. Sources: OECD 2011b; World Bank 2008, 2013d. a. Depending on the sample of benchmark countries, the increase could be measured as 54 or 55 percent. In terms of definition, the report stated only that “[c]ountry systems and procedures typically include, but are not restricted to, national arrangements and procedures for public financial management, accounting, auditing, procurement, result frameworks and monitoring.” b. The Program for Results, as it is commonly known, built in turn on the Bank’s Sector-wide Approaches and its output-based lending, both of which necessarily make larger use of countries own systems. See World Bank 2013f for a discussion. The African Development Bank (AfDB was similarly concerned that the “use of country systems creates additional risks of delays, poor procurement and inadequate 20 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT financial reporting” (AfDB 2011). It adopted a cautious approach to the use of country systems through a three-phase program. Phase I is based on the understanding that the use of country procurement systems is acceptable for all non-ICB contracts for goods and works if a country’s systems for domestic procurement are judged acceptable and provide for the progressive use of these systems. The AfDB has done a rigorous assessment for about 45 countries in Africa (representing approximately 97 percent of annual average approvals) and found their procedures (regulatory and institutional systems) acceptable for Phase I. AfDB does not require a change or waiver to existing policies (procurement guidelines), as it relies on existing provisions allowing use of NCB procedures, similar to those found in the Bank Guidelines. As fiduciary responsibility remains ultimately with AfDB, in Phase I AfDB applies its complaints redress policy. Phase II of the program moves to full use of national procedures, extending reliance on the regulatory and institutional systems of the country to include the judicial and complaint redress systems. Although these two phases are limited to non-ICB contracts, they will cover a majority of contracts financed by AfDB. Phase III will extend the program to full use of country systems for all contracts, including ICB. Implementation is not expected in the near term. The Inter-American Development Bank’s (IDB) guide for use of country systems lays out its approach (IDB 2010). It describes the criteria and procedures by which IDB assesses a country’s procurement system related to acceptability, conditions of use on IDB-financed contracts, and required improvements, if any. This program applies only to non-ICB procurement of contracts that are below the ICB threshold. A country is approved to use its systems through a waiver to IDB policies. 30 Accordingly, no changes to its policies were required. Country systems are assessed against MAPS requirements. Further requirements are added, where necessary, to elaborate IDB-specific needs (for example, where MAPS requires annual procurement plans, IDB elaborates plan contents, such as budget, schedule and method of procurement). The IDB’s stated objective is to use a country’s procurement system to contribute to project outcomes. It recognizes that the system may not meet all MAPS or IDB-specific requirements and may still be used, albeit with limitations. Moreover, it offers avenues for partial compliance and a progressive approach to increasing broader use. Noted flexibilities include: • It is not necessary to assess or approve the entire country system— subsystems or components may be approved. For example, approval might be limited to administrative entities such as national government agencies, state- owned enterprises, sectors such as infrastructure or health, or a particular state agency. It could also be limited to particular types of procurement, such as goods 21 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT and works or consulting services, or to particular procurement methods such as competitive tendering, but not other methods such as direct contracting. • It is possible to approve a system or component of a system if the intent of the requirement is met through other means. For example, a country may not have the required arbitration law, but other acceptable legislation or regulations may provide a legally valid tool for resolving contractual disputes. • Where a mandatory requirement is not met, it can be accepted conditionally. For example, if proper advertising is not mandatory under a country’s system but is done anyway, the system can be approved conditional on the practice being made mandatory, with the conditionality attached to the loan agreement. • Where nonapproval runs counter to the objective of contributing to project outcomes, a system may be approved with an attached requirement to address a deficiency. For example, if a required information system is seen as substandard, approval can be granted on the understanding that the information system is upgraded, possibly with IDB assistance. The IDB program of use of country is progressing. IDB informed IEG that as of July 2013, four countries (Brazil, Chile, Ecuador, and Mexico) had been approved for advanced use, and eight countries are awaiting approval for partial use. 31 The IDB and AfDB use of country systems programs contrast with the Bank’s pilot, which strove for full compliance to a set of criteria for the full range of procurement circumstances, including ICB (except consultant recruitment). Furthermore, with the exception of the assessment of India’s state-owned Powergrid, the Bank’s intention was to approve national systems for use on all projects in a country. IDB and AfDB avoided the “all-or-nothing” approach, striving to gradually move to full use of country systems though a measured process involving partial use, conditional acceptance, agreed improvements, and considerations of intent and objectives. Donors told IEG that the Bank led the MDBs in the endeavor to use country systems, that its methodology was detailed and rigorous, and that it evolved through extensive discussion with a number of stakeholders. However, its pilot appeared to many to set an overly high bar. This meant that in countries where official development assistance was not a significant portion of government expenditure, there was little incentive to engage in time-consuming dialogue and to possibly adjust systems to comply with Bank requirements. Such countries may have thought it more pragmatic to continue using their own systems for government expenditure and to ring-fence systems for Bank-financed projects. In contrast, countries where official development assisstance is high are typically those with weak capacity. Perhaps knowing that their systems had very little chance of being accepted, they had limited incentive to join the Bank’s pilot. 22 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT It was further observed that the Bank’s approach to the use of country systems had to balance conflicting interests. Although some countries felt that criteria were overly strict, the private sector in many developed countries was skeptical of the regulatory and judicial systems of many developing countries—they believed that their chances of winning international tenders would be lower if country systems were used. Donor countries gave mixed signals. It was not uncommon to see the bilateral arm of a country pushing for increased use of country systems, although its representatives to the MDBs advised caution. USE OF COUNTRY SYSTEMS—STAKEHOLDERS’ VIEWS ON THE WORLD BANK PILOT Among the countries evaluated for the Bank's pilot, three (Morocco, the Philippines, and Senegal) were part of the 11-country survey conducted as part of this review, although none fully qualified for the program in the end. Several other surveyed countries explored the possibility of participating in the pilots but did not. The reasons given, in case study countries, varied: • Bangladesh and Ethiopia did not think they were ready or thought that the requirements would be too difficult to achieve. Azerbaijan was judged by a recent CPAR as not ready. • Indonesia said that the pilot methodology was flawed and that it was too mechanistic and would not lead to a mutually acceptable harmonized system. • Two countries had other priorities at the time. Mexico wanted to focus on addressing recent CPAR recommendations, and Turkey wanted to focus on meeting European Union requirements. 32 Among the pilot countries, Morocco and the Philippines commented that requirements were too stringent—in the Philippines the four or five items of noncompliance were seen as minor or easily ring-fenced. Development partners in the Philippines further reported that, although the government had seen the pilot as a valuable exercise, it was disappointed by the outcome. The government invested significant financial and human resources and had an expectation of successfully moving toward full use of country systems, but the final decision-making process was opaque, and it was never fully explained why the Philippines did not pass Stage II. Morocco, the Philippines, and Senegal all said that the use of country systems should not be an all-or-nothing process, that it is highly possible to take incremental steps toward greater use of country systems even if countries could not adjust their systems in their entirety. If the objective of the pilot was the identification of countries where loans could be made that would be implemented entirely through country systems, the Bank’s approach was too onerous and detailed to be practical. However, it illustrated a 23 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT number of areas in which country systems do fall short of Bank standards and that would require some compromise in terms of the underlying principles of Bank procurement. Not all were of equal importance, and in some cases the issues that were identified could also be interpreted as country priorities (preferences for domestic providers, regions, specific suppliers). Other differences reflect countries’ use of modes of procurement that are little used in the Bank. The discussion shows that there is little consensus on the meaning of the use of country systems. In practice it appears that there is a continuum of use of country systems, rather than an absolute presence or absence of such use. USE OF COUNTRY SYSTEMS—STAKEHOLDERS’ VIEWS ON THE ADOPTION OF COUNTRY SYSTEMS Beyond the discussion of the Bank’s approach to its pilot, IEG reviewed a broad spectrum of stakeholder views on the more general theme of moving toward the use of country systems in Bank procurement. IEG found prominent differences between views of borrowers (government procurement offices, government counterparts, and implementing agencies) and private suppliers. Government Client Views From the perspective of most government clients, moving toward country systems has potential advantages (Box 1.6). First, it can ease the need to learn duplicate procurement process for implementing agencies. Learning one set of rules and processes instead of two saves human resources. Second, it eases layering in clearances, hence reducing delay. In some countries, national procurement offices oversee public procurement contracts and also give clearance. This leads to the need for two sets of “no objections” for executing agencies. Such clearance and oversight may also be delegated from a national office to a financial intermediary or other financing agent, which leads to similar layering of “no objections.” Third, it is pointed out that thanks in no small measure to support by the Bank over preceding decades, in many client countries the procurement regime is already very similar to that of the Bank. Yet there were caveats. Country systems themselves may not be homogenous, especially in federal governments, as pointed out in Mexico and Ethiopia. Although the majority of Bank-financed procurement occurs at the federal level, the use of country systems ultimately must take into account state regulations. Ideally, federal and state procurement legislation should be harmonized but typically the federal level has no jurisdiction over state procurement. In Mexico, among 32 states, only 2 apply federal procurement legislation. Ethiopia’s 13 provinces each have their own procurement regimes. Some clients perceive that suppliers are more responsive to tenders under Bank guidelines, especially in ICB, and there is the perception that 24 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT maintaining some procurement under the mantle of the Bank would give a competitive advantage. Box 1.6. Using Country Systems—Benefits Perceived by Client Implementing Agencies Especially in countries where Bank investment lending has been a declining part of its support, as in Mexico, even institutions well versed in national procurement legislation have to learn a new “procurement language” for a very small part of the procurement they perform, leading to inevitable confusion and delay. In Morocco, a Bank client starting its first project financed by the Bank mentioned that the necessity to mobilize eight people dedicated to a Bank project within its organization and to train them to apply the Bank’s procurement system represents a heavy burden. Bangladesh Bank staff noted that using country systems would reduce the requirement for procurement agents to learn two systems. Similar concerns were echoed in Ethiopia, Senegal, and Tanzania. Tanzanian clients pointed out that capacity constraints in terms of learning two systems are compounded in projects with several implementing agencies (an example was a project involving 18 municipalities and 6 sectors). Requirements to obtain no objections from the government tender board and from the Bank added significantly to processing time. Senegal also noted that the double prior review process adds time and cost. In Indonesia, implementing agencies expressed support for greater use of country systems to reduce confusion and limit procedural delays. Source: IEG procurement country surveys. Private Sector and Supplier/Contractor Views Contrary to views of Bank country staff and government stakeholders, private sector contractors and suppliers clearly favored the use of Bank systems over country systems, as they perceived them to have embedded safeguards. Suppliers point out that despite frustration with the implementation processes of the Bank, the Bank Guidelines are broadly acceptable (albeit with some areas where modifications could be useful, as discussed earlier in this chapter) and help maintain a level playing field. Reasons private agents offered included greater transparency and integrity, more timely payment, and effective recourse (Box 1.7). Some suppliers also claimed greater administrative burdens in government processes. Box 1.7. Using Country Systems—Drawbacks Perceived by the Private Sector In Morocco, the private sector expressed confidence in the Bank’s procurement system, saying it provided assurance of timely payment, transparency and integrity, and a recourse mechanism. Companies emphasized that the Moroccan system lacked an independent complaints procedure. In Peru, the predominant view was that efficiency, economy, and 25 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT transparency would suffer if Peru’s procedures were applied. Similarly, in Turkey, private firms, including national consulting firms, expressed their preference for the Bank’s procurement system and believed local systems overemphasize price. They consider the administrative requirements of the national system to be burdensome (especially for small firms); procedures are not always transparent; and the handling of complaints protracted, with dispute resolutions lasting years. There was also the issue of debarred firms that particularly affected national consultants; compared to the limited number of firms debarred under the Bank, national lists are very lengthy, as they include firms debarred for performance reasons, such as delays. Local consultants prefer to bid on contracts that fall in sectors outside the scope of the Public Procurement Law – such as the defense sector. International consultants reported abstaining from participating in competitions conducted under Turkish regulations. The private sector in Senegal also did not favor the full use of country procurement systems, citing the lack of transparency (especially in the light of the changes made to the scope of the procurement code by the government in 2011) and risk of fraud and corruption because of the lack of an independent and efficient complaint mechanism and court system. There was also concern about local audit capacity and the extent to which it could effectively ensure the integrity of the system. Although contracts are between a supplier and the government and therefore should not be affected, there is a perception that the Bank has powers of persuasion when its own systems are applied, which may be reduced if country systems are used. Source: IEG procurement country surveys. Conclusion—Partial Use of Country Systems Countries questioned whether the issue is Bank systems versus country systems per se or the implementation of such systems. In several countries, the transition to the country system is not considered difficult, as the differences are typically minor. The more critical challenge is ensuring integrity, minimizing interference, and achieving governance outcomes—the principal issue is reliance on country governance systems rather than focusing on procurement systems (identified as being the procurement procedures and rules). In Bangladesh, stakeholders stressed the importance of Bank oversight and its contribution to contract formulation and execution; they also saw it as reducing political interference and bribery. Greater clarity is needed on the extent to which the Bank would be able to maintain oversight in the procurement process, if country systems are to be used. Currently, Bank Guidelines/bidding documents/contracts give the Bank the right to inspect and audit. Bank management has maintained that Bank legal remedies—for example, declaring misprocurement—would still be available, and also that existing rights and 26 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT obligations with regard to fraud and corruption would not be altered. Yet it is not clear how complaints, dispute resolution, or observance of sanctions lists would operate, or whether the Bank’s role in moral suasion, for example, to ensure timely payment, could still be exercised. Given also that in practice many countries have multiple systems, in many areas—as has already occurred with NCB—the Bank could adopt a more flexible approach, making incrementally greater use of country systems, though continuing to work with countries as necessary to uphold best practice. In particular, the Bank could scrutinize the areas of difference that are observed systematically across different countries and prioritize those that could benefit from more flexible approaches by the Bank, compared to those where, on the basis of the Bank’s principles, further dialogue with countries and further capacity building may be desirable. Upstream and Downstream Involvement in Procurement ADEQUACY OF UPSTREAM INVOLVEMENT One of the propositions explored in Bank management’s preliminary vision of a new procurement policy is the need for greater Bank involvement in both upstream and downstream aspects of the procurement process. IEG’s exploration of stakeholder views concerning the need for greater involvement first reviewed perceptions of current levels of Bank upstream involvement, in the planning, design, and procurement preparation phase. Bank upstream engagement was positively perceived in most cases, especially in terms of the provision of advice on the preparation of terms of reference, technical specifications, and the use of the Bank’s standard bidding documents (Table 1.4). 33 More involvement would have been welcomed in some instances (for example, in Azerbaijan, because of the limited capacity of some project implementing units and consequent difficulties preparing technical specifications, drafting bid documents, and so forth). Such problems were compounded in complex areas such as information and communications technology (ICT) (Azerbaijan and Turkey). There were some client references to limited Bank staff capability. These frequently referred as much to task team leaders as to procurement staff. In Ethiopia, task team leaders were sometimes perceived as having limited technical knowledge and unable to support clients in the writing of technical specifications needed for bidding documents. 27 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Table 1.4. Bank Support During Procurement Design and Planning Proc Bank Country Question Avg staff TTLs clients In terms of project preparation, how effective is the Bank's procurement design 3.4 3.2 3.4 3.5 /planning process at contributing to overall project outcomes? With respect to the Bank's requirement that the borrower prepare a procurement plan 3.6 3.4 3.8 3.6 that is updated regularly throughout the project, to what extent does this requirement contribute to procurement outcomes? Source: IEG questionnaire. Note: 1 = negligible; 2 = modest; 3 = substantial and 4 = high. TTL = task team leader. Upstream involvement at the acquisitions phase need not be only in terms of design details for individual contract packages. It refers also to the entire procurement planning process, and at an even higher level, the integration of the procurement function in overall budget management and the public finance architecture, where, so far, the Bank’s role has been limited. 34 DOWNSTREAM INVOLVEMENT—ROLE IN CONTRACT MANAGEMENT Some clients voiced concerns that current Bank involvement in contract management may not be enough, especially because of limited country capacity to appropriately supervise contract management. In the Philippines, Bank procurement staff pointed out that procurement constitutes “asset acquisition”; for this reason, they felt that procurement input is required beyond contract award and up to the point that works are turned over or goods are delivered and installed. In Bangladesh, procurement staff noted the need to support contract management if there are payment issues, contract termination issues, and the like. Support was expected from Bank task team leaders, Bank technical experts, or consultants, as much as from procurement staff. Clients attributed limited Bank engagement (in Bangladesh, Ethiopia, Peru, Tanzania, and Turkey) to limited task team leader time, skill constraints, frequent rotation, and sometimes inadequate engagement. Bank task team leader supervision was deemed ‘’spotty” or “nominal” at best and in some countries was seen as restricted to semiannual review missions. Limited team leader contributions were thought to partly reflect limited incentives for more task team leader engagement in downstream supervision aspects, especially those involving procurement. Although there is a procurement accreditation option for team leaders, interest is limited because of the lack of rewards or incentives. There is some perception that accreditation may conversely lead to increased responsibilities as well as increased exposure to risk, with limited additional returns. 28 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Bank support to implementing agencies in the postcontract award period was considered necessary to seriously combat fraud and corruption in project implementation. In Indonesia, where fiduciary risks were a continuing matter of concern, there is a view that although risks during the procurement process can be addressed to some extent, risks during contract management remain. The scope for large-scale fraud and corruption during project implementation is exacerbated by inadequate record maintenance, poor tracking of construction quality, absence of quality assurance plans, and inappropriate contract variations, which may be caused by incomplete design, inappropriate specification, delays in processing payments, or collusive practices. It was also claimed that when procurement ends with contract award, limited integration of internal controls implies that much important information pertaining to contract management (such as actual quantities of goods delivered, actual amounts paid, change orders, revisions, cancellations, and so forth) is not captured, resulting in limited knowledge of actual results. Bank procurement staff point toward a need for guidelines to staff on contract management, to ensure adequate controls and avoid reputational risks (Table 1.5). The Bank could prepare project management manuals to help executing agencies or require them to develop suitable procedures. Mention was also made in some countries, such as Bangladesh, that extending the period of engagement to upstream and downstream phases needed to be considered in tandem with adjustments to the nature of Bank engagement. The Bank could take a more mentoring than clearing role, with more hands-on engagement. Table 1.5. Contract Management and Implementation Country Bank proc Bank Question Avg mgmt staff TTLs Once a contract is awarded, is there still a need/role for a procurement staff or 2.8 2.9 3.1 2.4 accredited staff to oversee contract management /implementation? Or are TTL skills adequate? Source: IEG questionnaire. Note: 4 =s trong need; 1 = negligible need. TTL = task team leader. These discussions reflect some of the tensions concerning the role of the procurement function of the Bank in downstream areas of procurement implementation. Although there clearly are perceived needs for such support, it is not clear whether or to what extent the support should be provided by Bank procurement staff, Bank task team leaders, or technical experts, or by national institutions that can provide support; it is also unclear whether hands-on (project implementation agencies) are preferred or through oversight functions (such as a national audit office). 29 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Among Bank staff, there is a need to review the present incentive structure and to consider the nature of the engagement as much as its duration. In addition, there are legal issues. Contracts are made between a borrower/implementation agency and contractor, and the Bank is not a party to this. Thus, the role and extent of the Bank’s procurement function will need to be considered relative to the roles of other agents in the country. In Bangladesh, for example, procurement staff opined that more engagement with contract management represents an oversight and advisory function for national execution, rather than a transfer of responsibility to the Bank. Most other countries and international documents do not address contract performance or consider it a part of procurement. 35 The Bank could also fortify its support to countries (currently offered on a limited scale through the World Bank Institute [WBI]) to develop civil society oversight of public procurement and contract execution. Developing Domestic Suppliers and Offering Domestic Preferences PRESENT PROVISIONS FOR DOMESTIC PREFERENCES As stated in the Bank Guidelines, for decades the Bank has seen procurement as a tool to encourage the development of domestic contracting and manufacturing. It is one of four considerations that guide current procurement rules and procedures. Although provisions for domestic preference are built in to the guidelines, they allow domestic preference only for ICB and only for goods and civil works. For goods, borrowers may give a price preference of 15 percent to domestically manufactured goods. 36 For civil works, in qualifying countries (those with per capita income of $1.25 or less), domestic contractors may be given a bid preference of 7.5 percent over foreign bidders. Analysis undertaken by the Bank’s procurement anchor found that domestic preference provisions are seldom used, and when they are used, they seldom determine the outcome of bidding (World Bank 2012o). From FY99 to FY09, this provision was used on 0.3 percent of contracts (0.4 percent by value) and was estimated to have potentially affected the outcome of the bidding on only 12 of 57,000 contracts. Bank management points out that the use of domestic preferences is at the discretion of the borrower and the executing agency and may not always be invoked because of its potential for inflation of the bid price. A background paper (World Bank 2012o) concludes that it is doubtful that the domestic preference policy continues to benefit local industries; it also points out that any preferential 30 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT treatment, by definition, discriminates among suppliers, which goes against open competition and has the potential to be divisive. Rather, the Bank could focus on helping governments to encourage domestic industry and develop mechanisms, like e-procurement, to enhance transparency and fairness and lower costs for all bidders. Capacity building efforts thus become even more paramount. BROADER INTERPRETATION OF DEVELOPING DOMESTIC SUPPLY CAPABILITY IEG’s 11-country survey polled stakeholders on the usefulness of domestic preferences to client countries. Among Bank management, results varied by country, with more developed countries considering this less important. Domestic preference, as provided in the Guidelines, was considered substantially or highly justified in poorer countries: Ethiopia, Indonesia, and Tanzania. In contrast, justification was considered low in more advanced countries such as Peru and Turkey. Bank procurement staff in the field and task team leaders see some form of domestic preference as worthwhile, even though it is little used because of a sense that there are sometimes developmental and/or project-specific benefits to be derived from favoring local contractors and manufacturers. Clients point out that the Bank’s current domestic preference scheme appears ineffective, as it is narrow in scope and thus limited in relevance. In Indonesia, attention was drawn to differences between the Bank’s domestic preference scheme and that found in national procedures. Bank Guidelines do not allow preferences under NCB or methods of procurement other than ICB, and bidding must be open to all qualified firms, whether local or foreign. By contrast, the Indonesian preference scheme excludes foreign bidders for contracts below a specified value ($5 million for works, $1 million for goods, and $0.5 million for consulting services). In other countries, contractors are ineligible under Bank policy because of the country income threshold, though they are eligible under national policies. Other aspects of domestic supply were discussed in Bangladesh, Ethiopia, Indonesia, Tanzania, and Turkey, although they were outside the purview of the Bank’s current domestic preference policy. With regard to consultant contracting, it is perceived that the Bank’s usual insistence on securing geographic diversity (that is, not more than two bidders per country in consultant contracting) is considered to limit good development outcomes in countries where there are strong local consultant capabilities. Respondents think that the geographic diversity requirement screens out capable local bidders and thus discriminates against domestic consultants. Foreign consultants often do not bid in markets with good domestic capability, as 31 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT they fear that their price will not be competitive. Unintended consequences are less total competition and the screening out of qualified domestic firms. Management points out that there are flexibilities for using alternative shortlisting methods, or for including national experts explicitly in the scoring process. But these appear to be little known and little used by Bank staff and clients. Another unintended consequence is that domestic firms sometimes associate with foreign firms to meet geographic requirements, when they could have expressed their interest directly, at lower cost, without the foreign partner. Ethiopia, Indonesia, Tanzania, and Turkey all provided examples of such cases. Domestic consultants that have won joint venture contracts in such situations maintain that they then undertake the majority of the subsequent work. Country respondents suggested that the Bank drop the two-firm per country limitation, allowing more domestic consulting firms to bid. They also favored dropping the six-firm limit on the short list, to allow for more overall competition. As many borrowers apply preference schemes within national systems (Box 1.8), the Bank needs to understand the objectives and mechanisms involved, especially as the Bank moves to greater use of country systems. A broader review of the issue of encouraging domestic suppliers would have merit, not limited to the scope of the current and narrowly defined domestic preference policy. It would appear that there could be scope for more broad-based support to local suppliers, while taking account of competition and open markets. Box 1.8. Perspectives on Domestic Preference Internationally, different instruments may guide governments in conducting public procurement, including provisions for domestic preference, although this is one of the more controversial aspects of procurement policy. In addition to the Bank Guidelines, the UNCITRAL Model Law allows evaluation criteria to include a margin of preference for the benefit of domestic suppliers or contractors or for domestically produced goods. It also allows for any other preference, if authorized or required by procurement regulations or other provisions of state law. However, the World Trade Organization’s Government Procurement Agreement and European Union Directives do not allow such preferences. They maintain the principles of nondiscrimination and equal treatment among members and thus do not allow preferences (with some very narrow exceptions). Both the Agreement and European Union Directives say that socioeconomic considerations cannot be used as 32 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT barriers to restrict access to bidders of member states. As reported in a background paper prepared by the Bank as input to its Procurement Policy Review (World Bank 2012o), many countries pursue domestic preference objectives (such as development of small and medium-size enterprises), though their procurement policies vary. These findings are based on a survey of the experiences of the European Union and other countries (Australia, Canada, Chile, Japan, Malaysia, Mexico, South Africa, the United Kingdom, and the United States). The greatest number of formal preference and technical assistance programs are in Australia, Canada, Malaysia, South Africa, and the United States; they target women, indigenous populations, racial and ethnic minority populations, disabled populations, and small and medium-size enteprises. The tools used include explicit preferences such as set-asides for specific groups of vendors, award preference during the bidding process, and locale-specific targeting of preferences within specific regions of a country. Proposed new changes to the European Union Public Procurement Directive make several suggestions to promote small and medium-size enterprises, which could also be useful for the development of domestic suppliers: removing barriers such as turnover or years-of-experience requirements, simplifying bidding requirements (for example, allowing the use of self-declarations), packaging procurement to make it easier for small and medium-size enterprises to apply, and direct payments to subcontractors (Appendix A, last section). Sources: World Bank 2012o, IEG 2013. Environmentally Responsible Procurement—Potential for Bank Use There is no consensus on what environmentally responsible procurement—sometimes referred to as “sustainable procurement” or “green” procurement—constitutes. One Bank background paper defines green procurement as "the purchasing of products that provide environmental and related socioeconomic benefits" (World Bank 2012f). This broad definition would suggest any procurement practices geared toward limiting pollution or toxicity, recycling, increasing energy efficiency, protecting biodiversity, assessing carbon footprints, and possibly more when considering social benefits. The background paper reports life-cycle analysis as a way of capturing environmental costs and benefits through the procurement process, adding that a rigorous methodology for tracking overall fiscal costs is lacking. IEG points out that it is especially difficult where choosing the greenest product does not lead to the lowest price, that is, where the choice cannot be quantified or justified 33 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT monetarily. Ecolabeling can be used to specify goods that meet certain criteria, but problems exist with agreed standards. 37 Although green procurement is very much discussed, there appear to be challenges in definition and methodology and thus in developing commonly accepted policies and practice. BANK POLICIES, PRACTICE, AND EXPERIENCE The Bank does not have an explicit policy on green procurement. Bank Guidelines do not mention it. Indirectly, however, environment-related provisions can be found in the Bank’s standard bidding documents. Evaluation criteria provide for bid price adjustments based on life cycle costing, for example. Such criteria can be used to specify energy efficiency requirements and could be extended to other “green” specifications. The form of contract based on International Federation of Consulting Engineers (FIDIC) contracts includes a clause on protection of the environment. It reiterates that contractors must comply with environment-related technical specifications and are further bound by local laws (that may require environmental clearances, for example). These measures provide for inclusion of environmental concerns in the procurement process in practice, although without any guiding policy requiring strategic use of procurement to meet environmental or other socioeconomic objectives. IEG's 11-country survey found limited views on the topic. In 10 countries, Bank procurement staff responded to a question on how effectively the Bank’s procurement regulations and practices contribute to government policies related to green procurement. None said it was high, though in three countries (Azerbaijan, Bangladesh, and Tanzania) staff said it was substantial. The remaining seven responding countries said it was modest, low, or negligible. As the Bank has no explicit policy on green procurement, IEG assumes that these views derive from practices, perhaps related to requirements for environmental impact assessments or to bidding and contract provisions noted above. In the Philippines, the only country where clients raised the topic, it was said that green procurement was a policy area that the government wanted to pursue but that the Bank had shown little initiative. The Bank's procurement practices accommodate some of what is seen as necessary for effective procurement of environmentally friendly services and products. Put differently, although there are areas in need of strengthening (public-private partnership [PPP] contracting, for example), there is nothing in the Bank's policies or practices that prohibits employing life cycle costing or using environmentally efficiency technical specifications. Box 1.9 provides examples of Bank projects where 34 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT such factors are incorporated. Bank practice accepts imposition of national environmental laws in technical specifications and contract terms and conditions (FIDIC based). This relates mainly to regulations on mitigating environmental damage during construction. It is not clear if it would extend to imposing environmentally friendly regulations. IEG suggests more specificity in Bank policies (procurement or environment) on how national regulations are to be treated in this regard in Bank-funded procurement. Box 1.9. Procurement and Energy-Efficient Products and Services Two Bank publications evaluate the role of procurement to address energy efficiency—the first deals with obtaining energy efficient services, the second with obtaining energy efficient products. The first (World Bank 2010g) focuses on contracting with private sector energy service providers. Issues raised, primarily related to long-term contracting for privately provided services, are similar to those found in PPP procurement (Chapter 2); IEG supports adjustments that help achieve better outcomes. With respect to procuring energy-efficient products, a later publication (World Bank 2012f) recommends greater use of labels or certification, preferably shown to be more economical through life-cycle cost analysis. Life-cycle costing is accommodated in current Bank procurement practices through the bid evaluation criteria found in the Bank’s standard bidding documents. However, both publications recommend explicit policies that mandate procurement of energy efficient goods and services, as found in many national jurisdictions (the U.S. General Service Administration Acquisition Regulations, for example). Bank policies do not do this, and do not provide guidance where procuring the greenest product does not lead to lowest cost. Specifying products based on energy-efficient labeling or certification is controversial, as it must consider the stature of the certifying authority. The Bank accepts internationally recognized certification requirements as a valid technical specification, but in many instances requires “or equivalent” to be added, to not bias procurement to a particular supplier or country of origin. UNCITRAL, the World Trade Organization Government Procurement Agreement, and European Union approaches provide guidance on working within internationally accepted norms that consider fairness, competition, and trade regulations. Sources: World Bank 2010g, 2012f. INTERNATIONAL PERSPECTIVES Different international instruments can guide governments in the conduct of public procurement, including provisions related to socioeconomic and environmental 35 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT concerns. 38 Both the Government Procurement Agreement of the World Trade Organization and European Union Directives allow for specific reference to environmental factors in technical specifications, evaluation criteria, and contract conditions. Because of their trade-facilitation orientation, the Government Procurement Agreement and European Union Directives say that socioeconomic considerations cannot be used as barriers to restrict access to bidders of member states. However, the UNCITRAL model law recognizes and accepts socioeconomic policies in national legislation. These are defined as “environmental, social, economic and other policies authorized or required by the procurement regulations or other provisions of law.” Reviewing the environmental policies and policy frameworks of some European Union countries (mainly Finland, Norway, and Sweden) and six other countries (Australia, Brazil, Canada, Chile, Mexico, and the United States), IEG notes that all countries have policies that attempt to use procurement to achieve energy efficiency and environmental protection objectives. Two features stand out: the use of environmental standards (ecolabeling) in technical specifications and incorporating green considerations into the procurement process via life-cycle cost/benefit analysis. These features maintain the principles of competition and contract award based on technical compliance and most economically advantageous offer, while at the same time using procurement strategically to address environmental policy objectives. Box 1.10. Asian Development Bank—Loan Agreement Provisions on the Environment The ADB guide(ADB 2007) requires that its loan agreements include the following clause: …The Borrower shall ensure that the Goods, Works, and Consulting Services procured under this Loan Agreement are produced/rendered in a responsible manner with a view to resource efficiency, waste minimization, and environmental considerations, as set out in the Environmental Management Plan, Initial Environmental Examination, or other Project-related documents, as applicable. In this connection, the Borrower and the Project Executing Agency shall furnish, or cause to be furnished, to ADB all such reports and information as ADB shall reasonably request to verify compliance…. Source: ADB 2007. The Bank was a member of a working group on environmentally responsible procurement, through the Heads of Procurement of the MDBs. The working group did not produce harmonized guidance, although the ADB proceeded to prepare a guide for ADB use (Box 1.10). 39 The ADB guide works within the existing policy framework- policies related to procurement and to environment. It does not propose policy 36 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT changes, but guides staff on how to introduce environmental considerations into the procurement process, within existing policies. Its two main points refer to the incorporation of environmental standards (ecolabeling) in technical specifications and the use of life cycle cost analysis to factor environmental benefits into bid evaluation. Thus it appears that the ADB has integrated environment-related procurement matters into its project design and formulation more explicitly than the Bank. Achieving Value for Money in Bank Procurement Achieving value for money is intended to be a cornerstone of the Bank’s new procurement policy. The Bank’s ongoing procurement review proposes a shift to value for money principles, although details as to what this would involve have still to be spelled out. IEG’s review of the application of value for money in other jurisdictions provides some guidance to factors that the Bank could take into account. Although there is no commonly accepted definition of value for money in public procurement, a typical one would be “what a government judges to be an optimal combination of quantity, quality, features and cost, expected over the whole of the project’s lifetime” (Burger and Hawkesworth 2011). This definition incorporates the widely accepted elements of quality, features, and life-cycle costs. Beyond this, value for moeny is frequently broadened to include social and environmental externalities or factors such as employment creation, support for vulnerable groups, or local content. 40 Such definitions are difficult. What is of value to one person or government may not be valuable for another. The core issue is to determine what bid evaluation factors may be taken into account; especially controversial is the matter of how to apply nonprice factors. Strategies to introduce value for money concepts to public procurement must consider a number of sometimes competing factors: • Effectiveness of markets to deliver desired results related to cost, quality, innovation, and risk, considering such things as capacity of the procuring agent and levels of competition • Trade policy, in particular its relation to international competition and preference schemes, and alignment of procurement policies and practices with internationally accepted norms • Public accountability requirements of transparency and fairness • Methods to assess life-cycle costs and benefits, especially when imputing environmental and socioeconomic considerations • Mechanisms to control fraud and corruption. 37 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Austrailia, the European Union, the United Kingdom, and the United States are four jurisdictions that have taken a long-term and systematic approach to integrate VfM concepts into their public procurement practices (Appendix A has details on the application of value for moeny in six public jurisdictions). 41 The U.S. “best value” approach also addresses the matter of incorporating non-price factors to achieve a best value outcome. The European Union’s proposed updates to its public procurement framework further emphasize the role of value for money (Appendix A) with more explicit instructions on life-cycle costing, as well as the incorporation of social, environmental, and labor laws. Additionally, the Asia-Pacific Economic Cooperation forum (which includes 21 Pacific Rim countries) has endorsed value for money concepts since 1999, and the International Atomic Energy Agency is one United Nations agency that has adopted these principles (IAEA 2011). IEG has reviewed the application of value for money in each of these jurisdictions (Appendix A). Based on IEG’s review of how it is applied in other jurisdictions and agencies, the following observations are offered on how the Bank’s current policies and practices align with value for money concepts as practiced by others, and where there is scope for modifications in practice, if this practice, in its narrow sense, is to be reflected. APPLYING VALUE FOR MONEY IN THE WORLD BANK Accommodation of Nonprice Factors in Procurement Decisions With value for money, as defined by the European Union, MEAT (Most Economically Advantageous Tenders) directives procurement may consider such factors as quality, technical merit, aesthetic and functional characteristics, environmental characteristics, running costs, cost-effectiveness, after-sales service and technical assistance, delivery date, and delivery period, in addition to price. These are typically evaluated through life-cycle costing and present value assessments of costs and benefits. The Bank’s procedures, which prescribe selection of the lowest evaluated bid, provide for this, and it is commonly done (Box 1.11). For example, technical specifications described in evaluation criteria can stipulate required environmental characteristics, and criteria can be developed to take into account operations and maintenance costs over the life of a facility. 38 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Box 1.11. Incorporation of Maintenance Costs and Life Cycle Costing in Bank Procurement Two projects in Turkey provide examples of the incorporation of certain value for money principles in present Bank procurement, including life-cycle costing for the evaluation of energy-efficiency technical specifications. • Supply and Installation of Submarine Power Cable: Technical specifications for energy losses and for five years servicing costs have been built into the financial evaluation criteria for the underwater cable, thus providing examples of how life-cycle efficiency and long-term maintenance costs have been taken into account. • Construction of Natural Gas Storage Facility: The project incorporates a 15- year life-cycle cost analysis by imputing the present value of operations and maintenance costs in the bid price. Source: World Bank procurement documents. Although the Bank’s policies and practices (as found in the standard bidding document) do not preclude this aspect of value for money, there is no explicit recommendation to exploit the potential benefits offered; stronger direction could be given to staff and borrowers to reflect the practice within the Bank’s procurement system. Expanded guidance is also required on how to incorporate factors into a quantifiable cost benefit analysis and further on how to accommodate nonprice factors. The Bank’s requirement to award contracts to qualified bidders that submit technically responsive bids and whose offered price is the lowest evaluated does not allow consideration of non-quantifiable costs and benefits. Generally speaking, for procurement of goods and works, bid evaluation factors must be quantifiable in monetary terms (an exception relates to ICT procurement, where weighting of technical offers means that contracts may be awarded to higher priced offers that are scored as technically “better”). Accordingly, technical offers are assessed for compliance, and among all compliant offers, the contract is awarded to the firm offering the lowest evaluated price. Evaluated price may consider various factors (operations and maintenance costs and energy-efficiency benefits, for example), but these factors must be financially quantifiable. Technical compliance provides further value for money opportunities, as does the process to qualify bidders, either in advance of bidding (prequalification) or as part of the bid evaluation process (postqualification). For example, it is common practice to require a supplier to have been in business for a specified number of years, have a minimum annual turnover, and be solvent. Qualification requirements may be further extended to the manufacturer (who is not necessarily the bidder). Common 39 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT examples include requiring that the goods must have been in manufacture for a specified number of years, perhaps at a specified volume, or that the factory that produces the goods must have an operational history, and so on. Such measures are effective at ensuring quality and mitigating supply chain risks. However, beyond these examples, the Bank’s requirements do not easily accommodate nonquantifiable criteria—for example, the accommodation of merit points or other scoring systems— where the assignment of weights given to different factors could be arguable and where there is increased need for judgment in bid evaluation. It is sometimes questioned if these procedural approaches can truly achieve desired value for money outcomes, especially in complex procurement. Such outcomes are implicitly assumed to follow from, first, an appropriate specification of contractual requirements and, second, from the prescribed bidding and selection process for the preferred supplier. Expecting contracting authorities to comprehensively specify all aspects of high-value, high-risk projects in advance of any selection process may not be realistic: “Contracting authorities that carry out particularly complex projects may without this being due to any fault on their part find it objectively impossible to define the means of satisfying their needs or of assessing what the market can offer” (Kruger 2009). As noted, some jurisdictions extend nonprice factors to preferences for certain bidders and products (domestic firms, locally produced goods, firms owned by ethnic minorities, and so forth). Although this broader definition of value for money is not discussed here, a summary of experience in with regards to certain broader aspects is provided in Appendix A. 40 2. Adapting to Evolving Needs As pointed out by Bank management, at the time of introduction of the Bank’s current procurement policy, Bank lending followed a different model, focusing largely on infrastructure financing (World Bank 2012c). Client countries’ financing needs have since changed, and so have patterns of Bank lending. Are the Bank’s present procurement modalities equipped to deal with the evolving pattern of its lending? This chapter selectively explores some new directions of lending that present particular challenges for procurement, community-driven development (CDD) lending, lending in fragile and conflict-affected situations (FCS), lending for the acquisition of ICT, and lending for PPPs. In each area, concerns have been raised about the extent to which present procurement modalities are appropriate. Overall, IEG finds that in CDD projects and FCS situations, Bank Guidelines and processes per se are not problematic. Difficulties appear to stem from limited capacity among country staff, limited use of permitted flexibilities, and, for FCS, difficulties in the overall enabling environment. But in the more complex and newer areas of ICT lending and PPPs, although the Bank has made recent efforts toward incorporating the complexities presented by these challenging new areas, much remains to be done to align Bank procurement with prevailing best practice. Procurement in Community Driven Development Projects • Almost a quarter of Bank projects have some CDD elements, according to present nomenclature, and these appear to have a higher incidence of procurement issues than other projects. • Looking more specifically at a subset of projects that actually deploy CDD methods—that is, community management of resources—procurement does not appear to be unusually problematic, in terms of difficulties in implementation. • Reported problems have frequently related to the non-CDD components of those projects, reflecting difficulties in procuring (centrally) the consulting services and technical assistance needed to support the implementation of the CDD components. • The Bank Guidelines appear to have appropriate flexibility for such projects. Bank front line staff have been timid in the use of flexibility, because of an internal culture of risk aversion. 41 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • Although there may be potential for fraud and corruption in CDD projects, in view of their scattered nature and associated difficulties in monitoring, there are also cases where inflexibilities applied in Bank processing have led to misplaced efforts at “compliance”—and subsequent allegations of fraud and corruption. THE CDD APPROACH AND THE CDD PORTFOLIO Community-driven development (CDD) is an approach to development that gives control over planning decisions and investment resources to community groups and local governments. CDD programs are thus perhaps the farthest, in some sense, from the ICB model for which Bank procurement was originally designed. 42 CDD programs operate on the principles of participatory governance, demand responsiveness, community empowerment, and downward accountability. Initiated during the 1990s to help address problems of low capacity and lack of mutual trust, especially in postconflict situations, the CDD approach has emerged as an important element of the Bank’s lending programs over the last decade, with more than a quarter of IBRD/IDA lending during FY02–11 making at least partial use of the CDD approach. 43 Several defining features of the CDD approach have important implications for procurement processes—the numerous, small-value subprojects, often in remote locations, and dispersed over large geographical areas and decentralized implementation through local communities with limited access and exposure to formal market institutions. To what extent are Bank procurement policies and procedures, designed for large value purchases and implementation through experienced agencies, able to adapt to this structure, or do they need to undergo significant customization to support efficient and cost-effective implementation of CDD projects? In this section, IEG reviews the effectiveness of the present system, in terms of the following: • Enabling delivery of intended results including inter alia strengthening of country capacity for such procurement Assuring transparency and efficiency in the use of project funds during implementation. The evaluation triangulates findings from four main data sources: (i) relevant Bank documentation related to procurement in CDDs; (ii) a comparative and comprehensive review of portfolio data; (iii) a detailed selective review of a purposive sample of 34 CDD projects; and (iv) semistructured interviews with task team leaders/procurement staff in the Bank on procurement-related issues in CDDs. 42 CHAPTER 2 ADAPTING TO EVOLVING NEEDS IEG’s portfolio review covered all evaluations with CDD elements over FY02–11. It reviewed overall outcome ratings as well as risk ratings in project/program documents as well as Implementation Status Reports (ISRs) and examined overall development outcomes in relation to project procurement performance for CDD and non-CDD projects. Its sample of in-depth review of 34 CDD projects was intended to lead to an understanding of the nature of procurement problems affecting implementation and outcomes of CDD projects. CDD LENDING AND BANK PROCUREMENT POLICIES The most noteworthy feature of CDD procurement is that, unlike other projects with many small contracts (for example, rural development projects implemented by government agencies and/or project management units), the primary procurement implementation responsibilities rest with the benefiting communities themselves. Often their role extends even further: they may contribute some resources toward implementation of subprojects (in cash or in kind) and they may also be the “contractor/supplier” for some of the works and services needed for the subprojects. These multiple roles, coupled with usually limited familiarity with formal fiduciary processes, pose the twin challenges of keeping the processes simple and within capacity constraints, while at the same time having adequate checks and balances to minimize fraud and corruption. Addressing these twin challenges has meant allowing further flexibility in procurement procedures applicable to CDD projects but with continuing attention to the imperatives of “economy, efficiency, transparency and fair competition.” 44 Detailed guidance issued to staff asks that the application of the guidelines be “scaled to the project and the community’s capacity” and “take into account local culture and norms” while assuring a “high degree of transparency and accountability” (World Bank 2012d). As a general rule, CDD procurement is expected to: • Be simple enough as to be understood and implemented by the community using simple, standardized documentation. • Be sufficiently transparent to promote real competition among suppliers and to facilitate control in the selection of contractors and use of funds. • Balance risks with efficiency considerations. Given the nature of CDD projects, ICB, NCB, or international shopping are seldom used (outside components managed by the central implementing units—for example, for overall project coordination and monitoring and evaluation). Procurement methods commonly used include local shopping, local bidding, direct contracting/off-the-shelf purchases, and community force accounts. Prior reviews 43 CHAPTER 2 ADAPTING TO EVOLVING NEEDS are rare, and postreviews, as with other projects, are done on a sample basis and are often outsourced to third parties. Beneficiaries participating in the management of subproject funds are expected to provide important peer control over the use of funds. IEG PORTFOLIO ANALYSIS—PERFORMANCE OF ALL PROJECTS WITH CDD ELEMENTS Extensive previous analyses of CDD implementation by the Bank and by IEG have mixed findings with regard to the presence of procurement issues (Appendix B, summary section). To investigate the extent to which CDD projects encounter procurement difficulties, IEG first undertook a review of Bank investment lending, comparing both procurement difficulties and project performance outcomes for CDD and non-CDD projects, to assess the extent to which there may be differences in the two groups. 45 The portfolio included a total of 2,044 distinct investment lending projects, of which 552 used the CDD approach at least to some degree. Of these, among the CDD group, as of December 31, 2012, 287 projects had been completed and closed; the remaining 265 were still under implementation. In the overall sample of CDD and non-CDD projects, performance and results for 501 projects have been validated by the IEG through its Implementation Completion and Results Reports. For ongoing projects, IEG used data on performance from staff self- reported Implementation Status Reports (ISRs). Data for all projects together suggest that performance on project procurement is closely associated with overall project success (as discussed in Volume 1, Chapter 1). An analysis of the subset of project with CDD elements (as defined by the CDD anchor) shows further that these relationships hold equally strongly for CDD projects (Appendix B, portfolio analysis). 46 The analysis also suggests that CDD projects appear to have a higher propensity for procurement problems than non-CDD projects, and the performance gap appears to have widened over the past five years (Figure 2.1). The apparent downward trend in procurement performance for CDD projects brings into question the effectiveness of efforts made in recent years to simplify and streamline procurement processes related to the CDDs. Outcomes are largely invariant to region, sector, or loan size. CDD procurement tends to be more problematic in countries with lower levels of governance—FCS countries and countries with low Country Policy and Institutional Assessment (CPIA) scores (3 or lower). Interestingly, additional financing projects with CDD components show fewer procurement problems and better outcomes—a result perhaps of the self-selection associated with additional financing and of the learning and capacity building embedded in them. Finally, CDD projects aimed at improving 44 CHAPTER 2 ADAPTING TO EVOLVING NEEDS local governance or service delivery are more prone to procurement problems than those with the primary objective of community empowerment or improving livelihood security (Figure 2.1 and Figure 2.2). 47 Figure 2.1. CDD and Other Projects—Performance in Low Governance Environments Source: IEG analysis. Note: Projects with an ISR performance rating of moderately unsatisfactory. IEG PROJECT REVIEWS—FINDINGS FROM A SUBSET OF PROJECTS WITH COMMUNITY CONTROL The portfolio-wide analysis was complemented by a review of project cycle documentation (Project Appraisal Documents, ISRs, and where available Implemenation and Completion Results Reports) for a sample of projects. Three considerations guided sample selection. First, the sample was restricted to the subset of CDD projects with community control and management of investment funds, because from the procurement perspective, the most significant differences between CDD and non-CDD projects arise when communities have the responsibility for managing the procurement and implementation of subprojects. According to the Social Development Department database, this was the case for 237 of the 552 so- called CDD projects approved by the Bank during FY02–11. 45 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Figure 2.2. Procurement Performance and Development Outcomes by Type of CDD Source: IEG analysis. Note: For closed projects only. In a second stage, the review focused on the subset of the sample in which procurement problems were manifest. Thus, projects obtained after the first and second stages of screening were further stratified into four groups based on their procurement performance ratings by staff in ISRs (Table 2.1). 48 Table 2.1. CDD Projects: ISR Procurement Performance Ratings and Overall Outcomes Group I: 26 projects Group II: 57 projects Procurement rating of Moderately Procurement rating of Moderately Unsatisfactory or worse for Satisfactory or better for procurement procurement performance in the last problem in the last ISR and a rating of MU ISR or below in one or more earlier ISRs Unresolved procurement problems Procurement problems at some stage during implementation that were eventually resolved. Group III: 16 (closed) projects Group IV: 131 remaining projects Procurement performance rated No procurement problem reported in any Moderately Satisfactory or above ISR. Outcomes moderately satisfactory or throughout implementation; but with better. unsatisfactory overall outcomes. Source: World Bank. Note: ISR = Implementation Status Report. To maximize learning, the review focused on Groups I–III. 49 In accordance with the rest of the evaluation, a regionally balanced sample was drawn, based on the focus countries for this evaluation, as described in Chapter 1. 50 The final sample included 46 CHAPTER 2 ADAPTING TO EVOLVING NEEDS 34 projects: 17 in Group I, 12 in Group 2, and 5 in Group III. Thus, by design this sample is not meant to be representative of the CDD portfolio as a whole, as it is focused on understanding the problematic segment of the portfolio. In Groups I and II the evaluation focused on understanding the nature of the procurement problems affecting implementation, their actual/potential impact on intended project outcomes, measures taken by the Bank to help resolve them, and lessons for minimizing these problems in future. In Group III, the evaluation focus was on understanding the role, if any, that procurement might have played towards the unsatisfactory outcomes of the respective projects. Where necessary and feasible, the findings from the documentary reviews were supplemented by discussions with the respective task teams. Appendix B (project reviews) provides the relevant extracts from documentation for the 34 projects together with project-specific summary judgments on the procurement aspects of these projects. The main systemic findings are as follows: • Contrary to the patterns emerging from the portfolio-wide data, procurement in projects using CDD guidelines does not appear to be unusually problematic. In several cases of reported problems in Groups I and II, the procurement problems actually related to the non-CDD components of those projects. 51 More commonly, the reported problems reflected difficulties in procuring (centrally) the consulting services and technical assistance needed to support the implementation of the CDD components. 52 Of the 29 projects reviewed in Groups II and III, difficulties in implementing the CDD guidelines were an issue in less than a third of the cases. Thus, rather than a CDD procurement guidelines issue, this is a project design issue. • Overall, procurement performance of community-managed components does not appear to be a major constraint to effectiveness of the CDD projects (see Box 2.1, for example). • Where CDD procurement became an issue, it had less to do with the rigidities of the guidelines per se and more to do with how they were interpreted and used. In some cases, the difficulties arose because the client country’s own guidelines (for example, in Brazil; see Box 2.2) were more restrictive than what was allowed in the respective loan agreements with the Bank. In most cases, however, difficulty arose because task team leaders were reluctant to use the flexibility available under the guidelines because of excessive risk aversion. For example, requiring three quotations for shopping is still the norm in many projects, despite alternative options available in situations where that is not the appropriate 47 CHAPTER 2 ADAPTING TO EVOLVING NEEDS solution. 53 Staff training and leadership from the Procurement Board in disseminating the spirit of the CDD guidelines may help in this regard. Box 2.1. Afghanistan: Emergency National Solidarity Project II The Afghanistan Emergency National Solidarity Project II CDD project (2007, $120 million) was launched in challenging circumstances, where challenges were anticipated and largely overcome. Procurement issues did not relate to CDD components. The project supported community-level governance and management of subprojects aimed at improving access to infrastructure services. Because the subprojects were widely scattered, the Bank hired a procurement oversight consultant who supervised the process and built in-house capacity at the implementing ministry. Facilitators were hired on performance-based contracts to assist with social mobilization, subproject preparation and implementation, and capacity building for communities in target districts. This was accompanied by simplification of procedures at the community level and continuous training. Results were positive: 71 percent of the implemented subprojects targeted rural areas, and 73 percent of targeted rural families experienced improved access to basic services like power, roads, drinking and irrigation water, and education. Subprojects implemented during the project period generated direct employment for the beneficiary families, equivalent to 10.6 million labor days. Residual procurement problems. One challenge was that financial management and procurement responsibilities were split between various entities. Procurement problems related to non-CDD components and required procurement by a central agency, for example, the hiring of consultants or contracting of classroom construction. A sample of 25 subprojects for Kabul Province was reviewed, and documentation for 19 did not indicate major deficiencies. Source: IEG. • Potential misuse of project funds through fraud and corruption is a risk noted in almost all projects, and they emphasize transparency through proactive disclosure and accountability to the communities as a remedial measure. Such abuse, if it occurs, could be a reputational risk to the continuing viability of the CDD approach. 54 Sound post-reviews remain important vehicles for assuring fiduciary probity through checks and balances. 55 However, much of the postreview work focused on procedural compliance and not on outputs and outcomes. • Effort devoted to Bank supervision of fiduciary aspects is of major concern to task team leaders. Many of them believe that in the wake of the India Detailed Implementation Review, a disproportionately large share of supervision in CDD 48 CHAPTER 2 ADAPTING TO EVOLVING NEEDS projects is going into monitoring and oversight of fiduciary aspects—at the cost of more substantive project design issues (World Bank 2007a). This reflects the current input-focused approach to oversight of fiduciary aspects in CDD projects seeking compliance with myriad processes in innumerable subprojects. Shifting the CDD paradigm so that communities are expected to deliver agreed outputs and outcomes rather than trying to control their procurement of the inputs may be a way around this dilemma. Box 2.2. Procurement Issues in the Maranhão Integrated Program Although the Maranhão Integrated Program (2004, $30 million) built on good experience from an earlier project, problems arose with procedural compliance. Post-review exercises in 2008 looked at 100 contracts for 77 community subprojects approved in 2007. Evidence was found of systematic misapplication of Bank procurement guidelines and red flags of possible fraud. Problems detected included invalid quotations or quotes that did not meet the requirements of agreed shopping procedures. According to the Implementation Completion Report, …concurrent Bank missions on financial management … noted that (i) these procurement irregularities were not related to financial management practices, and (ii) the investment subprojects identified in April 2008 physically had been completed to the satisfaction of the beneficiaries. In other words, problems were related specifically to the misapplication of procurement procedures. Closer examination revealed no evidence of systematic fraud. But procedures had been misapplied, apparently because the implementing agency and the communities did not understand them properly. In addition, there was misunderstanding on whether the Brazilian law or Bank procurement guidelines took precedence. The project was closed in December 2008 after the allegations of fraud added to allegations of corruption and subsequent political instability at the state and municipal level. Up to this point the project financed 626 community subprojects—short of the 1,200 targeted at appraisal—36 percent of the loan was already disbursed, and beneficiaries were satisfied with project results. Despite the good pace of delivery of results, the project outcome was rated Unsatisfactory, mainly because at the moment of cancellation, 64 percent of the loan had not been disbursed and because of the allegations of fraud during the procurement process. Source: IEG. 49 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Procurement in Fragile and Conflict Situations • Increasing Bank engagement with countries in FCSs, often as the fulcrum of multidonor trust funds, heightens Bank responsibility for effective procurement in these environments. • Although issues related to procurement are pointed out in a number of country strategy reports in FCSs, their frequency is not greater than similar issues raised for all countries. • The Bank offers flexibilities in procurement in such environments, many of which have been helpful and widely used. Conversely, such flexibilities have also been used in operations that are not under special procedures. • Many flexibilities afforded for FCS procurement—such as greater delegation to field staff, greater choice over procurement methods, or higher thresholds—are in line with Bank-wide reform proposals and consistent with the findings of the present report. • The need for special regimes for such operations and clients may diminish over time. Around 30–35 countries have been included each year within the scope of the Bank’s definition of FCS, including approximately 1.5 billion people. 56 These areas are plagued by fragility, instability, and conflict, as well as delayed economic growth. Given the particular vulnerabilities and numbers of people affected, the international community is paying close attention to these countries to find means of restoring or reinforcing economic progress, as well as to improve the mechanisms through which international organizations offer assistance (World Bank 2007c, 2011e; OECD 2011a). The Bank has become a focal point, not only on account of IBRD/IDA financial support, but because of its leveraged collaborations with bilateral and multilateral aid organizations. Bank collaborations have helped harmonize approaches to working in FCSs and mobilize additional resources, notably through multidonor trust funds administered by the Bank (IEG 2006b). Although IBRD/IDA commitments to FCS over the period FY03–12 (Figure 2.3) show no sustained trends, funding to FCS from Recipient Executed Trust Funds has been increasing both in absolute terms and relative to total commitments from such funds over the same time period (Figure 2.3). 57 Given that the Bank administers not only its own resources but also those contributed by other donors, its fiduciary role assumes increased importance. Bank policy toward FCS has aimed to balance fiduciary controls with an appreciation of the unique operating environment that often requires rapid response and operational flexibility in countries with low capacity levels. Because most lending to FCS occurs through the investment lending vehicle, 50 CHAPTER 2 ADAPTING TO EVOLVING NEEDS the efficiency, transparency, and effectiveness of Bank procurement processes assume increased importance. Figure 2.3. Trends in IBRD/IDA and RETF Support to FCS (2003–12) Source: IEG analysis. Note: FCS = fragile and conflict-affected stat; RETF = Recipient Executed Trust Funds ANALYSIS OF PROCUREMENT ISSUES IN FCS—ISSUES IDENTIFIED IN CASS IEG’s analysis of Country Assistance Strategies (CASs)/Interim Strategy Notes (ISN) began with a sample of CASs from 10 FCSs over FY06–11 to identify the nature and extent of procurement-related issues. 58 Around two-thirds (16 of 25) mention procurement problems that limit development effectiveness. Eleven of the 25 CAS reports reviewed had specific procurement-related monitoring indicators in the CAS matrix, and 6 identified procurement-related outcomes. The problems referred to stemmed mostly from capacity bottlenecks. It is interesting that IEG’s review did not highlight Bank procedures as being a major impediment to procurement in FCS (Box 2.3). Thus two key areas emerged: • Lack of procurement capacity—The most cited procurement problem, mentioned in 13 of 25 CASs. Low procurement implementation capacity is compounded by weak public administration systems, including flawed legal systems for procurement and difficulties with financial management that compound challenges in operating environments. • Integrity and transparency—Nine of the 25 CAS reports mentioned issues related to corruption in procurement. 51 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.3. Procurement Issues and Procurement Support in FCS—Evidence from Selected CASs Capacity issues are frequently mentioned. Four FCS CAS/ISN reports referred to problems in the legislative and regulatory framework related to procurement. In Liberia, “Weak procurement legal framework and practices lead to inefficiencies in the use of public funds.” Low levels of country capacity impact procurement capacity; for example, “Haiti’s financial management and procurement systems continue to show significant shortcomings, while institutional capacity is weak and corruption prevalent.” The following CAS/ISN reports also mention weak procurement capacity: Nepal (FY04 and FY09), Haiti (FY07), Liberia (FY09), Burundi (FY05), Lao PDR (FY05), and Sierra Leone (FY10). Integrity and transparency, specifically corruption have also been highlighted. For instance, in Nepal, “…one of the growing concerns is that public procurement processes are being undermined at times by acts of intimidation to deter competition.” Corruption and lack of transparency were also mentioned in the following reports: Timor Leste (FY06), Nepal (FY04), Haiti (FY05, FY07, and FY09), Burundi and Afghanistan (FY09). Planned CAS support—Procurement-related outcomes and monitoring indicators: • Eleven reports had procurement-related monitoring indicators in the CAS/ ISN matrices and six had procurement-related CAS outcomes. In some reports, the Bank’s planned support to procurement was included with other policy initiatives. The FY05 Sierra Leone CAS, for example, had both outcomes and monitoring indicators related to improving transparency, accountability, and efficiency of procurement operations in the country. • The Haiti, Lao PDR, Sierra Leone (FY05), Liberia, Nepal (FY09), and Sierra Leone (FY10) CAS/ ISN reports included both procurement-related outcomes and monitoring indicators in the country strategy matrix. The Afghanistan (FY06), Timor Leste (FY06 and FY10), and Burundi (FY02 and FY05) reports included only monitoring indicators but no outcomes. Source: IEG. Box 2.4. Procurement Guidelines under OP 8.0 and OP 10.0 OP 8.00 Procurement Guidelines OP 10.0 Procurement Guidelines Described procedures for designating emergency Not Mentioned staff—Rapid Response Committees with an assigned procurement staff member to provide fiduciary support. Project Preparation Facility thresholds increased Project Preparation Facility from $2 million to $5 million. thresholds increased from $5 52 CHAPTER 2 ADAPTING TO EVOLVING NEEDS million to $10 million. Retroactive financing limit is increased from 20 Unchanged percent to 40 percent. Allows for increased clearance thresholds (field Unchanged staff can approve no objections for projects that were previously only approved by RPMs). Allows for the borrower to contract with United Unchanged Nations agencies, nongovernmental organizations, regional institutions, and other multilateral/bilateral donors to undertake project implementation—including procurement. Provides for simplified procurement arrangements Unchanged offering greater flexibility, speed, and simplicity by delegating procurement review and clearance authority to task teams and allowing methods other than NCB or ICB. Refers to the Fiduciary Principles Accord an Unchanged agreement between the Bank and the United Nations that allows agencies that are signatories to rely on its own fiduciary requirements when implementing trust funds administered by any other signatory Provides for a standard form of agreement for Unchanged supply of health related goods through the United Nations, United Nations Provision of technical services. Allows for procurement agents and project management agents. Increases the threshold to $500,000 subject to review Requires Consultant and clearance by RPM for CQS, describes Qualification Selection appropriate use of Direct Contracting in FCS. contracts greater than Shopping threshold of $500,000 for goods and $300,000 be cleared by RPMs $1,000,000 for simple works. Unchanged Shortlist of consultants where fewer than six firms have expressed interest and where a wide geographic spread of shortlisted consultants cannot be achieved. Provides for the use of Positive Lists, the reduction Unchanged of commodity customs and the use of Force Account. Additionally, OP 8.50 recommends that Unchanged 53 CHAPTER 2 ADAPTING TO EVOLVING NEEDS procurement teams working in FCS consider the following: Taking a sequenced approach, allowing low risk items to move ahead; Increasing prior review thresholds—dependent on the circumstances; Using a monitoring and/or procuring agent; providing for preshipment inspection especially for commodities; and intensifying Bank supervision. Source: World Bank. Note:CQS = Consultatnt’s Qualifications Selection; FCS = fragile and conflict-affected state; ICB = international competitive bidding; NCB = national competitive bidding; NGO = nongovernmental organization; RPM = regional procurement manager. Though significant, these numbers should be viewed in perspective. IEG’s overall review of country CASs, to gauge the extent to which procurement-related issues are identified (undertaken in Chapter 2 of Volume 1), highlighted similar proportions of CASs which mentioned procurement related issues, and in these, too, capacity- building issues were the primary concern (Box 2.4). ANALYSIS OF PROCUREMENT ISSUES IN FCS – REVIEW OF PROJECT DOCUMENTS IEG next undertook an analysis of Project Appraisal Documents based on a sample of 58 Bank projects selected from 10 FCSs between FY06 and FY11; IEG sought to assess the extent and the manner of utilization of procurement flexibilities under OP 8.0 in World Bank operations in FCSs, to gauge their usefulness and to see whether any supplemental measures may be needed to increase procurement performance in these environments. 59 For each country, IEG selected at least four projects that represented lending under both emergency and normal processes. 60 To identify which projects were more likely to be emergency projects, IEG used the following criteria: • Projects with a stated objective that was emergency related OR • Projects explicitly identified as emergency recovery loans. Findings from this analysis were primarily that: • The use of procurement flexibilities as laid out in OP 8.0 in FCSs was not limited to emergency projects. • When OP 8.0 flexibilities are used, strong justification is rarely provided. • The most frequently used flexibility was simplified procurement arrangements such as shopping, single-source selection, and community participation in procurement. 54 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • Procurement agents were used in countries with severe capacity constraints regardless of whether the projects were considered emergency projects. • Most of the project ISRs did not include discussions of the procurement flexibilities applied during project implementation. ANALYSIS OF PROCUREMENT ISSUES IN FCSS—STAFF QUESTIONNAIRE In addition to examining project-level information, IEG sought to incorporate feedback from Bank procurement staff with extensive experience in FCSs. IEG administered a questionnaire to 15 procurement specialists and hub coordinators who, combined, have experience in at least 45 such countries or territories. The questionnaire asked respondents about procurement in FCS, including but not limited to the following topics: the extent to which lending operations used OP 8.0, whether Bank staff used the flexibilities allowed or avoided risk; the nature of the specific streamlined procurement procedures that are most frequently utilized; the impact of OP 8.0 on economy and efficiency; impacts of borrower procurement capacity; and Bank engagement with other donors and multidonor trust funds. Overall, respondents point to extremely low or scarce procurement capacity; low private sector capacity in parallel; and lack of technical expertise and difficulty in attracting personnel, including Bank procurement specialists to relocate to FCS. 61 Low levels of market competition and limited bidder interest—especially bidders from outside the country—are also pointed out. 62 Most respondents did not think that delays in implementation were caused by procurement issues, and most respondents thought that the Bank’s processes in FCS were at least substantially efficient. Extent to Which OP 11.0 or OP 8.0, Respectively, Are Used in FCS The extent to which OP 8.0 was used in FCS was mixed. For example, although South Sudan conducted all operations under OP 8.0, Cameroon exclusively used OP 11.0. This has been the practice in the West Bank and Gaza too, but for a short period. In Nigeria, only multidonor trust fund projects were prepared under OP 8.0, whereas in Timor Leste, only one project was undertaken using OP 8.0. Notwithstanding their preparation under OP 8.0, projects in some FCS (the Democratic Republic of the Congo, Guinea Bissau, and Liberia) rarely took advantage of the flexibilities provided for in OP8.0 and instead applied the normal OP 11.0 processes. 55 CHAPTER 2 ADAPTING TO EVOLVING NEEDS OP 8.0 Flexibilities That Were Used the Most Among respondents, there was consensus on the frequent use and benefits of higher approval authority to procurement staff in the field, often up to regional procurement manager level, and greater use of higher thresholds for NCB and prior review; greater use of rapid procurement methods such as direct contracting (reportedly used most frequently); simple shopping or use of prequalified consultants; and greater use of extension of contracts. Respondents indicated that three countries—the Democratic Republic of the Congo, South Sudan, and Sudan— regularly used accelerated bidding and streamlined procedures and found them to be highly effective. However, there was a caution on higher NCB thresholds in low-capacity environments. In some cases “roving consultants,” provided by the Bank to troubleshoot, have usefully supplemented government capability through just-in- time support—for example, in Ethiopia and Sudan. It was also cautioned that higher thresholds under OP 8.0 (or its equivalent) are rendered irrelevant if all country thresholds are raised. It was also pointed out that putting senior and more experienced staff in place amounted effectively to raising thresholds. Respondents also pointed out that some provisions such as selection based on consultant qualifications and direct contracting are useful but that they don’t necessarily need an OP 8.0. Contract extensions were widely used in Iraq, South Sudan, and Timor Leste, for example. Also, in Timor Leste, single sourcing under other nonemergency projects was used widely, mainly through twinning arrangements with quasigovernment entities and universities in the region. Borrower concerns about perceptions and potential accusations of corruption arising from the use of single source selection partly explain why this flexibility was not put to use in FCSs such as Iraq. Finally, the use of United Nations agents in FCS operations was mentioned as one possible way of adding procurement resources, mainly by the procurement specialists from the Africa Region, but the results were mixed. Country Capacity Is the Most Limiting Factor Limited country capacity to implement procurement was cited as a major factor in 8 of 15 cases—most respondents indicated that the Bank tried to address this by providing more expert staff and resources in the field offices, with an emphasis on quality and experience, as opposed to numbers of staff. 56 CHAPTER 2 ADAPTING TO EVOLVING NEEDS One respondent indicated that country capacity was a more significant factor than prior review, methods, or clearance thresholds. Countries’ greatest need was the provision of guidance in the process, to the extent of helping clients prepare bidding documents or advising on bid evaluation. The Bank has tried to provide both better trained staff as well as training for client procurement staff. Difficulties persist in hiring and retaining well-trained local procurement staff. Risk Aversion Among Bank Staff Opinions on risk aversion among Bank staff are mixed. Some respondents indicated that Bank staff, task team leaders, and procurement specialists are perceived as inexperienced and risk averse and recommended more training; others indicated that there was no evidence of risk aversion. Two respondents indicated that the flexibilities, though available, have not been used because of staff risk aversion. Risk aversion is more evident among junior staff, who are hesitant to make decisions that require judgments; there is limited incentive for them to do this and considerable downside risk in the event of any subsequent issues. In Africa, extensive efforts have been made to increase the amount of Bank staff with substantial procurement experience. This has increased capacity on the Bank side of the procurement process. According to one respondent, thresholds were effectively raised by raising the seniority and decision making authority of Bank staff. Bank Engagement with Donors in Procurement in FCS Bank engagement with donors in FCSs is usually in the context of Bank-administered multidonor trust funds. One exception was the Democratic Republic of the Congo, there was no joint engagement between the Bank and other donors there, especially in the health sector, which had more than 20 actors. A meeting was called to try and remedy the situation. The result was the “Kinshasa agenda,” which was based on the Paris declaration and Accra workshop recommendations. Differences in approaches do not appear to have been an issue, as in such cases the Bank administers the totality of funds, under Bank Guidelines and processes. This has generally been acceptable to other donors. The Impact of OP 8.0 on Economy and Efficiency Respondents did not feel that the use of OP 8.0 measures led to a loss of competition and thus less economy. In FCS circumstances, often the presumption of competitive markets is not correct. In terms of efficiency, there was a general endorsement that the flexibilities, when used, added to efficiency in the contracting process. However, there is not always a clear distinction, in terms of circumstances of application and 57 CHAPTER 2 ADAPTING TO EVOLVING NEEDS eligibility, between OP 11.0 and OP 8.0. And the main overall issues remain: lack of country capacity and lack of sufficient private sector capability or interest. Summary and Conclusions Overall, IEG found that in fragile situations, procurement capacity among country counterpart staff may be a greater binding constraint than the Bank Guidelines. Although procurement flexibilities under OP 8.0 have been useful, most could also have been used without formal recourse to a separate set of guidelines. The documentation of permitted flexibilities is nevertheless useful, particularly for less experienced staff, who may have the greatest hesitation in taking risks. There is little incentive for risk taking and a substantial downside if decisions are erroneous. Increases in review thresholds (in line with the recommendations of this report) are under consideration in several Bank regions. In Africa it is claimed that some OP 8.0 procurement flexibilities are being rendered irrelevant by the raising of all thresholds (clearance, prior review, and methods thresholds) in FCS countries. The Latin America and the Caribbean Region is moving in a similar direction, currently developing new thresholds for Haiti. And one staff member noted that putting more senior or experienced personnel in the field has also been tantamount to raising thresholds. One surprising finding to emerge is the unevenness of the application of permitted flexibilities. Thus, many eligible projects have not used such flexibilities, and conversely, they have also been used in contexts where OP 8.0 has not been specifically invoked. Going forward, given the loose application of the regime so far, fine tuning of the flexibilities may not necessarily be needed; what would be useful is the ability to exercise discretion in drawing on them. Going forward, greater overall flexibility in Bank Guidelines, should this occur, is likely to reduce the need for resorting to special provisions. It is clear that the greatest constraint in FCS environments is not so much the Bank’s procurement policies, which do have provisions for needed flexibility, as implementing capacity constraints. Reinforcing capacity among country staff, but also raising awareness of Bank staff and supplementing that as appropriate with external resources, are priorities. Very recently, the Bank adopted a new policy directive towards procurement in Fragile States, as part of the investment lending simplification and reform (World Bank 2013e). While processes remain broadly similar to OP 8.0, the scope of applicability has been widened to include, beyond situations of urgency, situations of low capacity including, for example, small or remote countries. 58 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Procurement of Information and Communications Technology • Bank systems for information and communications technology (ICT) procurement have improved in the recent past, and the Bank Guidelines have permitted new flexibilities and clarifications that are well suited to the ICT process. Bank standard bidding documents for ICT projects, with their scoring process, Bank acceptance of two-stage bidding, and Bank approvals, when needed, of sole source, and its clarification of conflicts of interest for consultants, are welcome moves. • Legacy difficulties remain, reflecting the adaptation of the present system from one designed for a very different purpose. The Bank’s standard bidding documents are not user friendly and are not aligned with industry norms. • Although the system has flexibilities, many are discretionary and the complex chain of clearance for no objection can lead to significant delay. • Task team leaders and procurement staff have a limited awareness of available flexibilities and hesitate to use them, in part because of risk aversion, but also because of a limited understanding of their benefits. • Better project design is an issue, and this requires training and awareness building among the team leaders. • Country capacity is sometimes—though not always—a constraint. • Although the Bank has consistently offered advice and support for contract implementation, its interventions have been somewhat skewed toward the project planning, rather than implementation phase. THE BANK’S ICT PORTFOLIO AND THE SCOPE OF THE EVALUATION ICT procurement differs from other procurement of goods and works in that it typically encompasses a mixture of professional services and goods—supply of hardware and software, typically involving customized development of specialized applications. The unique challenges of ICT procurement derive from the complexity of the systems being procured, which continuously evolve and expand in parallel to changes in user needs and advances in technology. Moreover, such projects are typically implemented by agencies not knowledgeable in the relevant technical areas or in complex procurement (a central bank, for example). Many ICT projects are in the government administration and finance areas and have a broad stakeholder base that extends beyond the implementing agency. Decision making is further complicated as they typically involve change management that can disturb vested interests (tax reform, for example). The Bank’s ICT portfolio is significant (Figure 2.4), and ICT components are found in a range of sectors. International Finance Corporation (IFC) ICT projects have also grown significantly and now dominate the Bank portfolio in numbers. Complex information technology systems are frequently included within projects related to public administration 59 CHAPTER 2 ADAPTING TO EVOLVING NEEDS (budget management systems) or the financial sector (central bank systems) or in seemingly unrelated areas such as the rural sector (cadastral surveys) or the human development sector (health modernization, for example). IEG’s evaluation focuses on the effectiveness of the Bank’s procurement system for large and complex information technology systems, typically involving the procurement of hardware systems and customized software. 63 IEG’s evaluation draws on (i) a review of the Bank Guidelines and standard bidding documents for ICT procurement; (ii) structured interviews/surveys of procurement specialists and task team leaders, based on a sample of projects in the ICT portfolio; and (iii) semistructured interviews with a wider range of Bank staff with specific experience in ICT transactions, including information technology specialists from the Bank’s Transport, Water, Information, and Communications Technology unit, from the procurement anchor and the Bank’s legal department. 64 IEG also undertook a review of previous IEG, Quality Assurance Group, and Bank reports that refer to Bank procurement structures for ICT (Appendix Box B.1). Figure 2.4. World Bank Group: ICT Portfolio 2003–10 Sources: IEG ICT database, based on IEG (2011b). 60 CHAPTER 2 ADAPTING TO EVOLVING NEEDS ICT AND BANK PROCUREMENT PROCEDURES ICT procurement, as with all other procurement, is governed by the Bank Guidelines and one set of its standard bidding documents that is specially designed to meet the needs of ICT. 65 One main feature of the bidding documents for ICT procurement is that they provide, as an option, evaluation criteria that score the technical proposals of bidders and then combine the scores with prices to derive an overall best bid. This is unlike the standard bidding documents for procurement of goods and works, which specify award of the contract to the lowest evaluated priced offer among technically compliant bids. This feature recognizes that procurement of information technology systems involves not only procurement of hardware and software, but also technical services, which require an evaluation of technical capability as well. The information technology bidding documents give guidance on developing technical specifications tailored to ICT systems. The contract terms and conditions similarly address ICT-specific requirements, especially in the areas of intellectual property rights and software licensing. Although earlier versions of the guidelines included clauses that recognized the unique procurement requirements posed by projects involving ICT, in 2011 revisions were introduced to specifically address ICT-related procurement concerns: 66 • Two-stage bidding for procurement of complex systems. 67 The guidelines specifically mention ICT procurement as an area where rapid technological advances make it difficult to prepare technical specifications in advance of the bidding. In straightforward procurement of works, a step-by-step process is followed in which an engineering design is first completed to develop technical requirements. Bidding documents are then prepared and a contractor is selected, who completes the works according to the design. For an ICT system, the bidding process will likely turn up unanticipated requirements. During contract implementation, technology may change and new requirements surface, both leading to the need to adjust the contract. In such cases, the guidelines say, the Bank may require the borrower to use two- stage bidding (see Box 2.5). • Direct purchase for backward compatibility. 68 A particular problem for ICT procurement is that it is often necessary to procure equipment that is compatible with existing systems (Windows operating system, for example). The guidelines make it clear that for ICT systems, compatibility issues may require specification of a particular brand of equipment, but that in such instances, Bank no-objection is required. • Framework agreements. 69 These are long-term agreements that set out arrangements for ongoing procurement over the term of the agreement. Although ICT is not specifically mentioned in the clause, it is possible that 61 CHAPTER 2 ADAPTING TO EVOLVING NEEDS framework agreements could be used to procure, for example, simple software maintenance or servicing. Box 2.5. Two-Stage and Single-Stage Bidding The guidelines suggest that for complex ICT procurement, rapid technological advances can make it difficult to prepare complete technical specifications in advance of bidding. In such instances, two-stage biddng may be appropriate. Under two-stage bidding, un-priced technical proposals are invited in a first stage. These are typically based on a conceptual design or performance specifications. During the first-stage evaluation, bids are subjected to technical and commercial clarifications, and requirements are adjusted to incorporate information learned through the process. In the second stage, amended bidding documents are issued and bidders are requested to submit final technical and priced bids. An IEG portfolio review found reluctance among task team leaders to use two-stage bidding—in 15 projects across 10 countries it was used in 5 projects in 4 countries (Bangladesh, Egypt, Russia, and Vietnam). The task team leader of a large information technology tax administration project said that single-stage bidding, where technical and price proposals are submitted and evaluated in a single process, was seen as quicker and equally effective. Procurement specialists with ICT experience disagree, saying that the time associated with two-stage bidding can avoid the much longer process of rebidding when single-stage bidding fails. Moreover, two-stage bidding allows purchasers to learn about available technology and to align their technical requirements with the best solutions offered by the market. The specialists say that single-stage bidding is best suited to situations where the purchaser has the capacity to specify the technical requirements to the detail that allows bidders to price all required items. Source: IEG survey of ICT procurement. IEG STRUCTURED SURVEY: PROCUREMENT STAFF AND TASK TEAM LEADERS IN A SAMPLE OF ICT PROJECTS IEG undertook discussions, on the basis of a structured questionnaire, with procurement specialists and task team leaders who had experience with the procurement of information technology systems in 15 major projects where information technology accounted for a large part of project value. 70 The intent was to review recent projects, under implementation, for procurement-related issues. Sample characteristics, including IEG scores, are shown in Table 2.2, together with perceptions on the quality of Bank support. Information on procurement methods used is in Table 2.3. 62 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Table 2.2. IEG Review of ICT Procurement—Project Characteristics Quality of Bank support during: Preparation Implementation Amount ($ Approval Nature of project Country millions) (FY) Type Quality Type Quality Health insurance administration Egypt 75 2010 HQ/SC Medium none na Central bank payments Bangladesh 350 2009 HQ/SC nc HQ High Education research network Bangladesh 81 2009 HQ/SC nc SC nc Financial management Vietnam 60 2009 HQ High none na Flood forecasting and monitoring Poland 184 2007 HQ/SC High HQ/SC High Insurance administration Poland 89 2006 HQ/SC High HQ/SC High Financial Reporting and Audit Pakistan 84 2006 HQ/SC High Note 1 na Public service administration Uganda 70 2006 HQ/SC High HQ/SC High Local government administration Uganda 55 2008 HQ/SC High HQ/SC High Internet connectivity Romania 60 2006 HQ Medium SC nc Power system control ( SCADA) Turkey 66 2005 HQ/SC High none na Land registration Serbia 30 2004 HQ Medium none na Customs administration Russia 140 2003 HQ/SC High none na Tax administration Russia 100 2003 HQ nc none na Financial management Vietnam 54 2003 HQ/SC Medium HQ/SC Medium Source: IEG ICT procurement staff survey. Note: HQ = Bank HQ procurement specialist; SC = specialist consultant; nc = no comment provided; na=not applicable; High = support timely and competent; Medium = timely and competent support, acceptable but with noted deficiencies; Low = incompetent, unresponsive, and/or ineffective.Implementation did not proceed (procurement cancelled). HQ = headquarters. SCADA = Supervisory Control and Data Acquisition. Most Bank procurement specialist support was provided in the preparation phase (Box 2.6 and Box 2.7). More Bank staff assistance was provided to review bidding documents and the bid process (evaluation) than to assist with implementation, by a factor of more than two to one. Bank headquarters procurement specialists provided support to all 15 projects during the preparation stage, but only on 6 during implementation. Eleven projects also utilized technical specialist consultants to assist during project preparation, and 7 used such consultants for project implementation. Qualitative comments in the IEG survey are positive—procurement specialist assistance was rated from “adequate” and “sufficient” to “very helpful.” There were no reported cases of Bank support being unresponsive or lacking in technical expertise. 71 63 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.6. IEG Review of ICT Procurement—Project Support Provided The Bangladesh Central Bank strengthening project is cited by procurement specialists involved with the project as a positive example where Bank technical support was provided throughout the project, including both the preparation and implementation phases. After an initial procurement failure, a repackaging exercise delayed the information technology component by about two years. Subsequent legal problems with one package introduced further delays. However, the system was eventually procured and successfully implemented. Another positive example cited was the Vietnam Financial Management Reform Project. For this project, the implementing agency received strong support from headquarters procurement specialists, specialist consultants, and procurement specialists in the field. In another country, the consultant who was hired by the Bank to assist with project preparation was reportedly competent and effective, but was not allowed to later work for the government to assist with implementation, as this was seen as a conflict of interest. The problem reported here is possibly due to interpretation of the pre-2011 guidelines that indicated issues with conflict of interest if the same parties contributed to both preparation and implementation. The 2011 revisions to the guidelines make it clear that conflicts between services related to project preparation and subsequent works associated with project implementation only pertain to provision of non-consulting services. This is another recent change, which could be brought to the notice of task team leaders and procurement staff. Source: IEG. Bank procurement procedures are seen as adequate, but some potentially helpful features are not frequently used. The overall perception of those interviewed is that the Bank’s procurement system is effective, with deficiencies noted in only three projects. All but two projects made use of the point scoring system. Further observations on the effectiveness of the procurement system are offered below: • Prequalification was used in only two projects. This might be explained by a sometimes limited number of suppliers for complex information technology systems. 72 • There is a preference for single-stage bidding. Two-stage bidding was used in only five projects. • Country procedures are sometimes easier to use. On two projects in Poland it was commented that using country procedures would have been preferable, as the government and bidders were unfamiliar with the Bank’s procedures, leading to extensive clarifications. Similarly, in Vietnam, Bank procedures were seen as overly complicated and difficult for the procuring agency to use. 64 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Table 2.3. IEG Review of ICT Procurement—Reported Use of Bank Procurement Methods Nature of project Procurement methods Country Amount Approval PQ 2Stage IT SBD Scoring Effectiveness ($ millions) (FY) (perception) Health insurance Egypt 75 2010 ✓ ✓ ✓ ✓ High administration Central bank payments Bangladesh 350 2009 ✓ ✓ High Education research network Bangladesh 81 2009 ✓ ✓ High Financial management Vietnam 60 2009 ✓ ✓ ✓ No response Flood forecasting & monitoring Poland 184 2007 ✓ ✓ Medium Insurance administration Poland 89 2006 ✓ ✓ Medium Financial Reporting and Audit Pakistan 84 2006 ✓ ✓ High Public service administration Uganda 70 2006 ✓ ✓ ✓ High Local government Uganda 55 2008 ✓ ✓ High administration Internet connectivity Romania 60 2006 ✓ ✓ High Power system control Turkey 66 2005 ✓ ✓ High (SCADA) Land registration Serbia 30 2004 ✓ ✓ High Customs administration Russia 140 2003 ✓ ✓ ✓ High Tax administration Russia 100 2003 ✓ High Financial management Vietnam 54 2003 ✓ ✓ ✓ Medium Source: IEG ICT procurement staff survey. Note: PQ =p requalification used; 2Stage = two-stage bidding used on at least one contract; IT SBD = the Bank’s standard bidding documents for information technology used on at least one contract; scoring = combined technical and financial scoring used on at least one contract; High Effectiveness = Bank’s procurement methods fully acceptable with only minor concerns; Medium Effectiveness = Bank’s procurement methods acceptable but with noted deficiencies that negatively impacted procurement process. SCADA = Supervisory Control and Data Acquisition. The bidding process and bid award were seen as generally satisfactory. However, problems exist with low competition and delays in evaluating bids. Nonacceptances of bids for procedural matters were limited to one late bid and one nonconforming bid security. There were no instances of cancelled bidding or rebidding. 73 The level of complaints was typical of any procurement and was adequately dealt with. Clarifications during bid preparation were few and did not result in significant changes to technical specifications. Clarifications were minor or procedural, not affecting system specifications. There was only one reported instance of the need to negotiate contractual terms after contract award, and this related only to software licensing terms (Serbia). Contract award prices were usually close to estimates. For six of the nine projects on which price data are available, award prices were equal to or below the estimates—although they were higher than estimate on the other three projects. 65 CHAPTER 2 ADAPTING TO EVOLVING NEEDS The largest reported overrun was the Russian customs system, which was approximately 60 percent above the estimate. However, generally price increases during bidding do not appear to be an issue, and this is consistent with the point above that there were few reported changes to technical requirements during bidding. And misprocurements or complaints seem to be only as frequent as in a general selection of projects. There was one reported Integrity Vice Presidency (INT) investigation resulting from a complaint from a losing bidder in one of the surveyed projects. Perhaps more substantively: • Competition was frequently low. More than half of the respondents said that competition was weak, or reported fewer than three bidders. The explanations given were political instability in the country (Egypt), few firms capable of supplying the required systems (Serbia), and limited interest because of the nature of the procurement (expansion of an existing system in Turkey). These explanations are reasonable. 74 • Two projects showed significant delays in contract award (out of only five answers to this question). The two reported delays were significant (60 days in one instance and 6 months in the other). It was noted that the project that incurred the six-month delay (Vietnam Financial Management system, 2003 project) incurred a further four-year delay for the contract to be signed following contract award. This was attributed to changes in technical design and scope of work and possible political issues. Contract implementation in the majority of projects is satisfactory, although there are some significant implementation delays. Eight of twelve projects reported delays from minor to three years, while four reported no implementation delays. In a number of projects, implementation is ongoing—final results are not known. So far, seven projects report no price variations exceeding 10 percent of contract value, though four projects report variations exceeding 10 percent. The largest, 33 percent, was reported for the Vietnam Financial Management system (2003 project). Causes of delay include: • Coordination problems between different contracts for supply of hardware and software. • Start-up problems arising from organizational changes within the borrower, and frequent changes in the Bank’s task team leaders. • Changes to scope of work during implementation. The Bangladesh Research and Education system reported a delay in commencement of bidding of three years because of difficulty finalizing technical specifications. This has delayed the entire project but is attributable more to project design factors than to procurement. 66 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.7. Messages from the ICT Portfolio Review Sixteen procurement specialists and task team leaders, interviewed for the portfolio review offered their general views on challenges encountered in procuring ICT systems. They broadly agreed that delays derive from the technical complexity of the systems being procured, requiring longer times than procurement of less complex products and works. ICT systems more frequently encounter changes to technical design and scope of work. Lengthy preparation of technical specifications delays procurement. This is compounded by the low capacity of implementing agencies. The Bank needs to support projects with adequate technical resources, such as ICT-knowledgeable procurement consultants. Moreover, applications are often politically sensitive, adding possible political interference. Project design and implementation typically require broad support from a range of stakeholders on features and functionality. Inadequate stakeholder input leads to systems that do not meet requirements or expectations. This in turn leads to ongoing modifications to technical requirements. There are problems in the formulation of the procurement methods, especially where exceptional methods are involved, such as sole-source negotiation. Such methods are sometimes the most appropriate, but Bank approval is a lengthy process that can negatively impact project outcomes. Source: IEG survey. IEG SUPPLEMENTARY INTERVIEWS: ICT EXPERTS, PROCUREMENT LEGAL EXPERTS IEG undertook additional, semistructured interviews, primarily with senior procurement specialists and ICT staff. On the positive side, some experienced procurement staff hail the ICT bidding documents as highly flexible. There are variations within the documents to cover simple goods (printers and peripherals, for example), services (including service contracts and professional services under consulting procedures), and complex systems that involve a combination of hardware and services (including both two-stage and single-stage procedures). These give procuring agents a range of options to fit different circumstances and substantially align with industry practice. Nevertheless, a number of specific issues were raised in terms of the appropriateness of the Bank’s current ICT bidding documents (Box 2.8). • The documents are voluminous, complex, and sometimes intimidating to borrowers and Bank staff. Though larger suppliers probably understand them, smaller or newer suppliers might not. Suppliers complain of the difficulty in bidding on Bank contracts. 67 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • The terms and conditions of contract do not align with industry norms. Bank documents evolved from the infrastructure procurement documents (FIDIC based) and are not typical of industry practice. • The Bank does not easily allow some procedures that are commonly used in the private sector. Sole sourcing and direct or competitive negotiation are examples, especially where a purchaser may already use a particular technology or product and therefore must purchase the same technology for expansion projects (Box 2.8). 75 Framework contracting, where the client can purchase from a list of qualified contractors, is another. Box 2.8. Negotiation to Procure ICT Systems Greater use of negotiated contracting has been advocated (by the Transport, Water, Information, and Communications Technology unit, for example) as an effective procurement method and as more in line with industry practice. Direct negotiation with a single supplier (sole source) may be used when, for example, an expansion of an existing system is required. Many factors need to be considered, such as compatibility of equipment and training of staff. Although the guidelines provide for sole-source procurement (clause 3.7) specifically in such circumstances, Bank no-objection is required. Competitive negotiation involves simultaneous negotiation with more than one supplier; the process typically culminating with interested bidders submitting a “best and final” offer. The method is provided in the UNCITRAL model on procurement. The European Union rules provide for competitive dialogue, and this is commonly used by governments and the private sector. Some Bank staff advocate the adoption of one of these models as more efficient, albeit with transparency and fairness concerns. To a certain extent, negotiation of technical and commercial factors can be accomplished under the Bank’s two-stage bidding procedure (see Box 2.5), although negotiation is only allowed in the first stage, at which point prices are not known. The second stage (price proposal stage) must be conducted without negotiation. Source: IEG. IEG COUNTRY SURVEY FINDINGS—EXPERIENCE WITH ICT PROCUREMENT IEG field visits indicated that present Bank Guidelines for ICT procurement are problematic, with only a “modest” score from country clients and Bank procurement staff, in terms of effectiveness. In five IEG case study countries, there was experience with ICT procurement. The following views were noted: 68 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • In all five countries (Azerbaijan , Indonesia, the Philippines, Tanzania, and Turkey), procurement of ICT was said to cause delays that negatively impact achieving project outcomes. • Development of technical specification is a common problem. System requirements are fluid and difficult to manage. This is partly a procurement issue, as specifications are seldom fully understood when tender documents are initially being prepared. It is also a contract implementation issue, as requirements will inevitably change during implementation and flexible processes are needed to accommodate this. • The guidelines and standard bidding documents have limitations when it comes to procurement of complex systems. They are seen as effective for straightforward procurement, but less so for procurement of complex systems. Two-stage bidding is held out as a good option for procurement of complex ICT systems, as it gives added flexibility to enter into dialogue with bidders on technical requirements. However, in both Indonesia and Turkey, two-stage bidding was criticized for adding time to the procurement process and not adding value. • There is an excessive emphasis on competition, where this may be problematic. In Turkey, the task team leader thought that there was a good rational for direct contracting so as to obtain technology that fit with an existing system, but was unable to get it approved (Box 2.9). In Tanzania, it was also commented that sole sourcing should be given greater consideration. • Skill shortages are a common problem, for both Bank and implementing agency staff, especially in projects being implemented through local governments. Because they lack knowledge, task team leaders and Bank staff do not utilize the flexibility that the Bank’s procedures allow. Consultants can sometimes but not always help. In Indonesia, consultants were used extensively and Bank headquarters staff was heavily involved, yet the procurement failed. The project implementing unit accused the Bank of being partial to certain suppliers, requiring arbitration and further delaying procurement. • ICT systems typically involve change management with attendant political complications. In Indonesia, one of the projects involved tax reform, which threatened vested interests. Moreover, the implementing agency was not technical (as compared to an electric utility for example). In the view of the task team leader, these were the primary factors contributing to the failed procurement. It implies that the problems that arise during procurement are often rooted in project design and other factors. 69 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.9. Using Sole-Source Procurement for a System Upgrade Turkey provides an interesting example of when to use sole-source procurement, as well as the difficulties in obtaining Bank approval—but also, the value added that Bank advice can offer. Two projects were undertaken to expand and upgrade a supervisory control and data acquisition system by the operator of the electric power utility. The first project expanded the existing system. The implementing agency initially wanted to sole source the procurement to the supplier of the existing system. The Bank encouraged competitive bidding, arguing that other suppliers could provide the expansion equipment and that it would function technically with the existing system. The implementing agency agreed. The supplier of the original system won the bid, but at a price lower than the estimate. The competition was effective in obtaining a lower price. The second project is currently being implemented to upgrade the entire system. The Bank initially insisted on competitive bidding, but the implementing agency argued that a system upgrade affected the entire system operations. To change equipment to a new vendor would require extensive changes to operations including retraining of staff. The Operational Procurement Review Committee agreed to the sole-source procedure, but it took one year to get Bank approval even though procurement specialists and the task team leader supported it from the outset. The example highlights certain issues. First, it is difficult and time consuming to obtain exceptions to normal procedures, to the point of serious project delays, and even with widespread support for the exception among Bank staff. Second, there is a reluctance to trust the implementing agency, even a technically knowledgeable and competent one like an experienced power utility. It raises the question of whether the Bank has superior expertise to override or even advise the purchaser. Last, the example highlight’s the Bank’s focus on competitive bidding and the difficulty it has in approving noncompetitive procedures. Source: IEG analysis. CONCLUSIONS Based on the above evidence, IEG concludes that Bank systems for ICT procurement have improved in the recent past, for example, with regard to its ICT bidding documents and their scoring process, two-stage bidding, concessions to the need for sole source, and clarification of conflicts of interest for consultants. Yet difficulties remain, reflecting the adaptation of the present system from one designed for a very different purpose. As a result, the standard bidding documents remain cumbersome and confusing (to users and bidders), and the Bank is not aligned with industry norms. 70 CHAPTER 2 ADAPTING TO EVOLVING NEEDS A second set of issues arises from the need to better educate task team leaders and procurement staff to better understand and more effectively use the present system. Although the system has flexibilities, many are discretionary and the complex chain of clearance for no-objection can lead to significant delay. Country capacity is sometimes—though not always—a constraint. Better project design is also an issue. Finally, though the Bank has offered advice and support for contract implementation, its interventions have been somewhat skewed toward upstream transactions. The limited present standard bidding documents for ICT are noted by clients. The clients note that the Bank should undertake an independent study of the practices commonly used by industry and governments (including the European Union) to align the Bank’s documents with best practice. It is noted that European Union directives do not include standard bidding documents; however, the European Commission has developed some for its own use. The Multilateral Development Bank Heads of Procurement has developed harmonized model documents for ICT procurement, but the Bank has not adopted them. 76 Limitations in the possibilities for dialogue between bidders and procuring agents are also referred to. Although the Bank’s two-stage bidding procedure provides the opportunity for a dialog to adjust technical requirements and commercial terms in the first stage, during evaluation of non-priced technical proposals, this is not in the second stage, during evaluation of price proposals. However, the review found that two-stage bidding is not frequently used for procurement of complex ICT systems because of perceptions that benefits do not outweigh the longer time involved. To enhance opportunities for dialog, the Bank could review options for the adoption of competitive negotiation / competitive dialog processes, similar to that of the UNCITRAL model law, and European Union Directives, and subject to borrower capacity. Greater use of direct negotiation with preferred suppliers (sole source) might be used less restrictively where circumstances warrant. Also echoed here, to a greater degree, were process difficulties caused by rigidities in oversight. Obtaining Bank no-objections can be difficult and time consuming because of low approval thresholds and the “chain of command” approach. Mechanisms are needed to fast track no-objections and approvals of exceptions. 77 At lower levels, inexperience may explain, in part, a reported rigidity of oversight. Better incentive structures are needed to address the risk-averse culture. Task team leaders, consultants, and especially field procurement specialists and others involved in project design would benefit from training, including in options for complex ICT procurement—for example, the use of two-stage bidding, direct 71 CHAPTER 2 ADAPTING TO EVOLVING NEEDS contracting, management contracts, framework contracts, and so forth, with guidance on how to apply them. 78 Implementing agency capacity is frequently limited, and there is a need to take measures to build capacity of the implementing agency well ahead of procurement, ideally as part of an overall ICT capacity-building strategy, prior to or during project preparation. Priority could be given to those ministries or departments with a long- term outlook of implementing many ICT projects and with an established procurement function. In some cases, consultants could have an important role to play in agencies that are technically weak. Bank specialists’ advisory role can come into conflict with the Bank’s oversight role and the need to remain neutral—for example, in Indonesia, where it was reported that the borrower accused Bank headquarters staff of favoring certain suppliers. Good design leads to better project outcomes, more so for projects involving large and complex ICT systems. Unique aspects of ICT projects that can impact procurement include a plurality of shareholders, for example, a financial management system with links between treasury and audit. Synthesizing competing demands and handling political interests is challenging, but if not done properly at the design phase. Unusual procurement methods—for example, sole source—can be built in to the procurement plan. Appropriate contract packaging—keeping procurement packages to a manageable number, ensuring packages are independent of one another, and ensuring single responsibility by avoiding splitting hardware and software contracts—would be beneficial. The Bank could also examine the extent to which contracting for information technology services could be used. And during implementation, discipline is needed to freeze the design so as to avoid continuous design modifications that keep adding features and functions as the system is implemented. These problems are not unique to the Bank, and IEG recognizes that every procurement system struggles with these challenges in ICT procurement. Gartner Research reports that the success rate in large private sector information technology projects is about 50 percent, and Bank outcomes should be viewed with realism (Gartner Group 2012). Procurement in Public-Private Partnership Investments Traditionally the Bank has loaned to client countries’ governments to finance public sector projects. But today, even in areas of traditional lending such as infrastructure, government agencies partner with private sector parties, which contribute financial 72 CHAPTER 2 ADAPTING TO EVOLVING NEEDS resources to deliver a public service or asset. This helps leverage scarce resources and can also have the advantage of bringing market discipline to the project. Yet lending to the private sector poses procurement-related challenges. The private sector typically uses commercial market practices to enter into deals that are financially attractive and bankable. It is also risk averse—it is not only the expected profit but also the certainty of the profit that ensures a total return. Risk includes factors such as strategic business partnerships, supply chain reliability, confidentiality of commercially sensitive data, dealing with developer-initiated proposals, working with preestablished concessions, and guarding against conflicts of interest. Public sector procurement faces a different set of specific factors: public accountability, open competition, fair treatment of prospective partners, and minimized subjectivity. When the private sector uses public funds, including Bank lending, the challenge is to strike a balance between public sector principles and commercial market practices. To the extent that the Bank’s public sector-oriented procurement processes do not match the demands of public-private partnership (PPP) projects, this impedes uptake of PPP investments. In view of the need to leverage Bank resources in a world where private investment is a growing share of the total, better procurement systems for such situations are critical for the Bank. IEG finds that although there has been recent improvement, there significant hurdles remain. • The Bank’s recent draft on procurement in private sector projects and transactions (World Bank 2013a) still has areas that lack clarity, especially with regard to the scope and standards of the Bank’s review of private investor procedures; situations where the Bank enters late, and is not involved in the design (including the design of procurement processes); situations where the Bank might be a minority investor; on-lending arrangements; and unsolicited proposals. • The Bank is seen as slow and ill equipped to adapt to the procedures of others, especially where quick response is needed, for example, to round out a financing consortium. Unclear guidance (for example, on competitive procedures, unsolicited proposals) and arrangements that rely on “acceptability” and where deviations require "waivers" are not a good option and lead to a lack of predictability and delay. • In the same vein, the Bank does not adequately recognize commercial confidentiality concerns that may make large players unwilling to share some information with a minor player, if it is considered commercially sensitive. 73 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • Even competitively selected concessionaires and other private sector partners can enter into conflicted downstream agreements, and Bank guidance in this regard is unclear. • Bank policy, although very similar to those of other MDBs, is not fully aligned with European Union practices; one important difference is the degree to which negotiation is permitted to arrive at a PPP concession agreement. PUBLIC-PRIVATE PARTNERSHIPS IN THE BANK’S PORTFOLIO Figure 2.5 illustrates Bank activity in PPP projects over the past 11 fiscal years. 79 The number of projects has shown a trend increase of about 70 percent over the period, from 19 to 32 per year. Financing commitments rose substantially after 2008, from $2.7 billion, peaking at $7.2 billion in 2011, possibly in response to the global financial crisis; they have since dropped to $4.8 billion in 2012. The data understate the Bank’s PPP activity, as some of the Bank’s work is advisory, leading to viable projects, but without Bank financing. IEG was told that the Bank would likely do more PPP loans, but it is constrained in part by procurement requirements. Figure 2.5. World Bank PPP Projects—July 2002 to December 2011, by FY Source: IEG database. Note: PPP projects for FY10–12 were compiled by and under the guidance of Get PPP and expanded to include noninfrastructure projects. The PPP projects for FY 2002–09 were estimated by IEG. IEG’s evaluation is based on four elements: (i) a portfolio review of 46 PPP projects over the years FY04–12; 80 (ii) interviews with Bank procurement staff from all regions and the anchor, with specific experience in PPP transactions, the Bank Get PPP anchor, the legal department, and IFC staff; (iii) comparisons with other countries; and (iv) IEG’s 11-country field-based survey of procurement. 81 Although earlier versions of the Bank’s procurement guidelines included clauses that recognized the unique procurement requirements posed by projects involving the 74 CHAPTER 2 ADAPTING TO EVOLVING NEEDS private sector, in 2011 revisions were introduced to increase flexibility (World Bank 2011f). The revisions make explicit reference to PPP projects. The revised policies are less prescriptive, tending more toward agreeing to “acceptable” procedures, typically defined by the principles spelled out in Section I of the guidelines (open competition, quality and transparency). These revisions are reflected in clauses 3.13, 3.14, and 3.15, which modify clauses 3.12 and 3.13 of the previous version of the guidelines. • As in earlier versions of the guidelines, concessions, or similar forms of private sector arrangements, are ideally selected competitively. If they are, the concessionaire or entrepreneur may procure according to its procedures and need not follow Bank procedures. The former requirement to select the concession using ICB was, however, relaxed to the more general “open competitive bidding.” Also, consistent with earlier versions of the guidelines, where a concession is not selected competitively, procurement by the concessionaire or entrepreneur (called “downstream procurement”) is to be competitive. However, the former requirement to use the Bank’s ICB procedures was relaxed to include invited bids and for smaller valued contracts, “methods acceptable to the Bank.” Consulting services were also added to items that the Bank may finance, procured according to the Bank’s consultant guidelines. • A new clause in the Bank Guidelines provided for, on an exception basis, Bank financing of incumbent concessions (including state-owned) and for the concessionaire or entrepreneur to apply its procurement procedures where Bank determines they are “acceptable.” The clause applies certain guideline- specific conditions related to conflict of interest and arrangements of state- owned enterprises and reiterates guideline requirements to ensure fair competition, transparency, economy, quality, and efficiency. It also requires the Bank to conduct postreviews during implementation to assure that procedures remain acceptable and are followed. • As in earlier versions, the revised guidelines address situations where the Bank funds an intermediary that on-lends to a beneficiary; the overriding requirement for procurement is the use of well-established procurement methods and commercial practices “acceptable to the Bank.” Where the beneficiary is investing in a PPP project, revisions to the 2011 guidelines direct the reader to specific clauses related to PPP procurement (summarized above), which helps define “acceptability” in terms of awarding PPP concessions. However, the responsibility for assessing procurement arrangements and oversight is transferred to the intermediary, under terms spelled out in the project documentation. 75 CHAPTER 2 ADAPTING TO EVOLVING NEEDS In the recent revisions to the guidelines, circumstances related to financing ongoing operations (incumbent concessions) are addressed. In addition to the guidelines, guidance on what are considered “acceptable” procurement arrangements is provided in a Guidance Note of September 2010 (World Bank 2010c). Although the Guidance Note predates the issuance of the 2011 revised procurement guidelines, it anticipates the revisions. For example, an appendix details what is considered “open competitive procedures acceptable to the Bank.” Guidance is also given in such areas as dealing with unsolicited proposals and how to establish oversight when working though financial intermediaries to address the quick response aspect commonly found in PPP on-lending arrangements. The Bank plans to update the Guidance Note to more fully reflect the revised guidelines. In January 2012 the Operations Risk Management, Operations Policy and Country Services Vice Presidency issued a draft note to aid the ongoing procurement policy review with respect to private sector projects. The note covers PPP as one of four categories of private sector engagement in projects, the others being direct lending to private sector entities, financial intermediary lending, and guarantees. The note proposes ways to fully utilize current features of the Bank Guidelines—it does not propose changes to Bank policy. This is a valuable initiative to improve PPP procurement within the existing policy framework. 82 However, certain areas for concern remain: • The extent to which the Bank would review and approve procurement capabilities and procurement actions of private borrowers is not always clear, nor are the criteria for such review. A practical concern among procurement specialists in the field and among task team leaders, as found by the IEG review, is the lack of clarity about the extent to which they should get involved in the procurement practices of concessionaires or private sector parties. The approach proposed in the note does not address this. • The note does not address circumstances where the Bank is not able to exercise oversight and influence in a step-by-step procurement process. It does not address special circumstances, such as where the Bank is a minority player in a project, in which case the major investors may not agree to the Bank setting terms for procurement. Similarly, there are situations where the Bank comes into a project late, after all procurement arrangements are in place or some procurement completed. Furthermore, the Bank is sometimes asked to work under an umbrella arrangement, where it needs to approve numerous small investments quickly. 76 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • The note does not address sanctioning, debarment, and right-to-audit provisions, matters that the IEG review identified as needing attention, especially where Bank requirements come into conflict with local laws. • The note does not elaborate how the Bank’s approach differs from that of IFC or why this is so. For example, the Bank applies a different level of review of private borrower practices and retains a discretionary ability to review certain critical transactions. It would also be helpful to explain the extent to which the Bank differs from other IFIs such as the European Bank for Reconstruction and Development and the European Investment Bank, and how Bank policies deviate from those of the European Union, which is especially important for clients working under European Union directives. • The note proposes the formation of a Business Review Committee to bring consistency of approach, and provide better access to Bank experts. While this could be a positive move, care is recommended to not add another layer of review. IEG reviewed the extent to which current Bank proposals are in line with other MDBs’ or client countries’ national policies (details are in Appendix B). Not surprisingly, there is little variation in the PPP procurement policies of the MDBs, reflecting 10 years’ efforts to align guidelines. Yet there are noteworthy differences between the Bank and IFC. IFC does not have documented requirements regarding procurement in its PPP investments. It reviews a concession agreement (and key contracts in a typical transaction) as part of its overall assessment of credit, development, and reputational risks in the transaction, rather than as part of a self- standing procurement exercise. IFC takes a commercial approach, assessing if the PPP arrangement was the result of an open and competitive process; if prices are reasonable; if the project complies with local laws; if it is financially viable; and if the parties are reputable, competent, and financially sound. This reflects the commercial orientation of IFC. Unlike the World Bank Group, which separates private and public sector lending operations between two different organizations (IFC and IBRD), most other MDBs have an in-house private sector lending window. Their operations include departments dedicated to private sector lending, yet they manage to accommodate this private sector lending function within the terms of procurement policies that are almost identical to the Bank’s (details are discussed in Appendix Box B.2). IEG REVIEW OF EXPERIENCE IN BANK OPERATIONS The IEG portfolio desk review looked at 46 projects approved in an eight-year period, up to December 31, 2011. The intent was to review projects that were approved 77 CHAPTER 2 ADAPTING TO EVOLVING NEEDS relatively recently but that were under implementation, as these would exhibit procurement-related issues. IEG selected the sample from a list of 73 projects based on data from financial and private sector development and Get PPP. 83 Of those 73, 25 related to creating an enabling environment and pipeline development and were not reviewed. The remaining 46, which involved Bank investment, were subjected to the review. Almost half (22) were from Africa, whereas Latin America and the Caribbean and South Asia accounted for nine and seven, respectively. 84 East Asia and Pacific accounted for four. Europe and Central Asia and the Middle East and North Africa accounted for two each (Table 2.4). Table 2.4. IEG Sample of Bank PPP Projects Reviewed (by region FY04–12) FY Region AFR EAP ECA LCR MNA SAR Total 2004 1 1 2006 1 1 2008 1 1 2009 1 1 2010 6 1 5 1 2 15 2011 12 2 1 4 4 23 2012 2 1 1 4 Total 22 4 2 9 2 7 46 Percent 48 9 4 20 4 15 Source: IEG portfolio analysis. Note: PPP = public-private partnership. Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; MNA = Middle East and North Africa; SAR = South Asia. IEG’s main findings are summarized below. • The Bank is typically the lead financier, and it uses investment lending to finance government investment in a PPP. The portfolio review showed that virtually all the projects used investment lending. 85 In 31 of the projects (67 percent), the Bank was the lead financier, and in the remaining 15 (33 percent) the Bank provided between 8 and 50 percent of the financing. In 36 loans (78 percent) Bank lending financed government investment in the PPP. In only six loans did Bank lending go to a financial intermediary for on-lending. 86 Four loans were to a special purpose vehicle. IFC was involved in four projects, three as advisor and one as equity investor.87 • In the majority of these projects the Bank is involved from the beginning and sets the procurement arrangements. In 41 (89 percent) of the projects, work did not commence until Bank financing was approved. The Bank was a late entrant in less than 5 percent of projects reviewed. Only one had significant 78 CHAPTER 2 ADAPTING TO EVOLVING NEEDS prior investments to which the Bank had to adapt. In 44 projects (96 percent) the Bank was involved in the design and was able to establish procurement arrangements—in the remaining two, the Brazil N 2 O Emission Reduction Project and the Uruguay Wind Power Farm Project, the Bank had to adapt to prior established procedures. These low numbers likely reflect the problems that the Bank has in entering after procurement arrangements are in place— were it not so difficult, the Bank would probably be able to take on more of these projects. Table 2.5. Financing Structure of Bank PPP Projects Financing structure Projects % Investment loan to cover government investment 36 78 Loan to financial intermediary for onlending to PPP investments 6 13 Investment loan to special purchase vehicle 4 9 Total 46 Source: IEG portfolio analysis. Note: PPP = public-private partnership. • Procurement problems were reported in over a third of ISRs. Of the 21 projects in which ISRs reported procurement issues or delays, the issues were directly or indirectly related to it being a PPP project in 17 (37 percent of all projects). • There are a significant number of PPP concessions awarded to incumbent concessionaires. Eight projects (18 percent) involved incumbent concessionaires. Of these, three were not competitively selected. IEG’s country surveys included interviews with a range of stakeholders to obtain their views on the suitability of the Bank’s procurement procedures for PPP projects. On average, across countries, and across interlocutors, procurement provisions in Bank guidelines for PPPs were perceived as “modestly adequate,” with an average score of 2 out of 4. In 8 of the 11 countries surveyed, the Bank team had no PPP operational experience, although in some instances there is a high level of interest in PPP (Box 2.10). The following views were noted: • In the Morocco Ouarzazate Project, a 500 MW solar energy complex, the government observed that the Bank tried to adapt its standard bidding documents but that this proved unsuitable for the project. Washington-based Bank staff, rather than country procurement staff, advised the implementing agency, but they did not have appropriate experience to design bidding documents, instructions to bidders, or contract conditions. A procurement expert has been hired to help resolve these issues. 79 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • In Tanzania, the Bank assisted with developing a PPP policy, a PPP law, and regulations. A private company is in advanced stages of discussion with the government to invest in a water project. It is thought that the Bank’s requirements of competitive bidding, at a stage when the private partner is already in advanced discussions with a supplier, would not allow Bank financing. The company is hesitant to reopen the process and has suggested that the Bank evaluate the transaction on its merits to assess whether it meets the standard of a competitive outcome. The company has suggested that IFC could be approached, as the transaction could meet IFC’s standard of a competitive commercial deal, in terms of outcome (result, price). • In Bangladesh, concern was expressed over the Bank’s engagement with a power project. The Bank was approached to provide a relatively small share of the financing, but it entered the process when bids were already evaluated and ready for award, yet would have required a “retrofitting” to Bank Guidelines. • In Peru, there is a general view that the Bank’s current procurement procedures are not adequate to meet the needs of more sophisticated projects or the use of new lending modalities such as PPP. The Bank has no PPP portfolio in Peru. Furthermore, task team leaders were of the view that Bank procurement staff are not sufficiently conversant with PPP requirements. More generally, they were critical of the Bank’s insistence on competitive bidding, suggesting that an IFC approach, which looks at reasonableness of price and results, is more practical. They added that the Bank’s PPP procurement guidance is inadequate, particularly in such areas as contracting on the basis of lowest subsidy, or for service delivery contracts. There is also a general view that the Bank’s Guidelines do not accommodate many common modern approaches to PPP; for example, in the Philippines the Build- Operate-Transfer law mandates “Swiss challenge” procedures for all unsolicited PPP offers. In Tanzania, respondents pointed out that procurement of toll roads on a “least-subsidy basis” is now very common. Box 2.10. PPP—Missed Opportunities in Other Countries? In the majority of field visit countries the Bank had not initiated any PPP lending. Yet there is a prevailing view that PPP is important. The following comments were extracted from field visit reports: 80 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Indonesia: There is no experience with PPP among Bank staff (including task team leaders and procurement staff), government (including implementing agencies and regulatory staff), or civil society (including consultants and contractors). The government views PPP as promising. Philippines: The Bank is not investing in any PPP projects in the Philippines, and implementing agencies have no experience working with the Bank on PPP projects. The unsuitability of the Bank’s Guidelines for small-scale decentralized PPPs, especially those supported via on-lending from the Development Bank of Philippines, was noted. The Bank’s procurement guidelines were seen as too strict for the private sector and made on-lending uncompetitive with other PPP financing sources such as private banks. Yet there are a number of PPP initiatives that do not involve the Bank, in education, health, power, and infrastructure. Noted was the department of education initiative to construct 10,000 schools over the next three years under a build, lease, and transfer scheme, as well as a (Manyland) water concession supported by ADB, and a Manila water concession supported by IFC This has ADB support. The Bank’s PPP activities are limited to supporting services—consultancy to provide technical support to a local government to develop a PPP water treatment plant project, for example. This is support for technical design and specifications, not financial or procurement advice. Azerbaijan: The Bank is not investing in PPP projects in Azerbaijan, and implementing agencies have no experience working with the Bank on PPP projects. Bank management notes that there is no existing legal framework in the country for PPPs. PPP initiatives in power and urban transportation, which do not involve the Bank, have been proposed. Turkey: The Bank is not investing in PPP projects in Turkey. The Ministry of Development (formerly the state planning organization) has been struggling with drafts of its PPP law, attempting to make it compatible with European Union legislation. Although deemed to be finalized by June 2013, it has still to be enacted. Many implementing agencies have limitations that would affect the quality and processing of PPPs. Nevertheless, Turkey has launched an ambitious 10-year PPP program and is currently managing a number of large PPPs, independent of Bank involvement, with 3 major infrastructure PPPs signed over the past year. Source: IEG field visits. CONCLUSIONS There is consensus that PPP is an important instrument for the Bank, though it is also widely thought that the Bank’s public sector-oriented procurement processes do not 81 CHAPTER 2 ADAPTING TO EVOLVING NEEDS match the demands of PPP projects. The guidelines and guidance notes tend to assume that the PPP project will be stand alone, that the Bank will be a major player, and that the Bank will be involved in the design of procurement processes from the beginning. This impedes uptake of PPP investments, especially in areas such as gap financing or situations where the procurement arrangements have either been put in place earlier, where the Bank is a minority or late contributor, or where the Bank operates through a financial intermediary. The Bank is seen as ill equipped to adapt to the procedures of others. Additionally, the Bank may need to respond quickly, to round out a financing consortium for example. Here, too, it is seen as unable to respond quickly to assess procurement arrangements. This is the experience of the India Infrastructure Finance Company Limited Project (see Box 2.11). Although current Bank policies, such as those introduced through recent revisions to the guidelines and articulated in the Bank’s guidance note, provide added flexibility, there is a need for more clarity in regard to review of “acceptable” procedures. Recent revisions may have shifted difficulties from the rules themselves to how they are interpreted and applied. Much now depends on how intent individuals may be to "do what they can to make it work." Box 2.11. India—A Tale of Two PPPs Although India was not included in the country surveys, IEG contacted India-based procurement staff, given the Bank’s considerable experience with PPP financing in India, and discussed their experiences related to two projects: (i) the Karnataka State Highways Improvement Project (KSHIP), providing Bank financing to the government for highway construction and operation concessions; and (ii) the India Infrastructure Finance Company Limited (IIFCL) project, where the Bank finances a government facility that provides gap financing in infrastructure projects, on an on- lending basis to IIFCL, a minority cofinancier. The investment reviewed was for a power transmission line concession. The two projects offer different experiences. With KSHIP, the Bank has had a long- term involvement and more than a decade of learning and adaptation. In the case of IIFCL, a new agency, the Bank is a minority player. IIFCL financing is limited to 20 percent at most of the value of a project (a “gap financing” model), and it typically relies on the lead financier for due diligence. For the IIFCL project, the Bank provided retroactive financing. Power Grid, though highly experienced at procurement and familiar with Bank’s procedures, used its own procedures, not knowing of the Bank’s participation at the time of procurement. Lessons learned 82 CHAPTER 2 ADAPTING TO EVOLVING NEEDS from these projects are: • Aligning government documents with Bank requirements is difficult, but adds value. In KSHIP, the implementing agency used model documents (RFQ, request for proposal, model concession agreement) issued by the government planning commission. Documents were redrafted to comply with Bank policy. Bank procurement staff feel that the implementing agency benefitted from adjusting to Bank requirements (for example, by securing the trust of bidders), although, unlike for investment projects, the Bank does not have standard documentation or a standard concession agreement for PPP procurement. • Getting involved early is important. In both IIFCL and KSHIP, staff emphasized the importance of Bank involvement in the design of processes, development of transaction documents, and building implementing agency capacity, at an early stage. • Clarify that procurement provisions specific to fraud and corruption (including right to audit) are “non-negotiable.” The government agreed to incorporate Bank policies in the procurement process and made the necessary amendments. In the case of IIFCL, this was done subsequent to the contract via a side letter. The Bank also insisted that Bank-sanctioned firms be barred from bidding but would not agree to allow use of India’s sanction list. Some disagreements between the Bank and implementing agencies relate to good practice, for example, in the handling of complaints, dispute resolution, and arbitration. Implementing agencies required bidders to submit a declaration that they would not challenge the outcome, as a condition of bidding. The Bank, appropriately, required proper complaints, dispute resolution, and arbitration systems. Other areas of disagreement are unique to Bank requirements and derive from its guidelines. For example, the Bank objected to eligibility criteria that invoke national security concerns (possibly giving preference to domestic companies), the involvement of state-owned entrerprises, and joint ventures of bidders with the implementing agency (Power Grid). National preferences, national security, and contracting with state-owned enterprises (including joint ventures) are common in many countries (see Chapter 3). Other guideline-specific disagreements arose regarding the implementing agency limiting the number of prequalified bidders and its use of two-envelope bidding (standard practice in India, and in many other countries, but not allowed by the Bank Guidelines), although eventually the Bank agreed. These issues are general to all procurement and not PPP specific, but illustrate the difficulties of imposing Bank policies into a new environment that involves parties other than government, where 83 CHAPTER 2 ADAPTING TO EVOLVING NEEDS the Bank does not necessarily drive the process. A management update was provided in June 2013 on the ongoing restructuring of the IIFCL Project, downsizing the loan amount from $1.195 billion to $195 million. New flexibilities have been added and the Bank has now approved a risk-based approach toward procurement due diligence. IIFCL is now allowed to follow a structured framework assessment of concessioning authorities that looks into various aspects such as governance structure and procedures for prequalification, bidding, and contracting. The Bank will not carry out a prior review of subprojects or IIFCL’s due diligence. These steps will make IIFCL’s due diligence of subprojects more streamlined and efficient. The Bank’s fiduciary safeguards will be to carry out a comprehensive annual audit of subprojects; a pre-assessment of IIFCL to ensure that it has the capacity to discharge its responsibility as agreed and a periodic capacity assessment to ascertain the established capacity is sustained over time; agreement with IIFCL on the necessary next steps to enhance its capacity to the level consistent with its new roles and responsibilities; and the right to carry out an Integrity Audit of the Developer, as done by IFC, to ascertain that the principles of procurement are adhered to in the selection of the developer. Source: IEG field visits. Although there are no standard approaches to procurement in PPP projects, there is scope for more tailor-made model procurement arrangements. In India, the lack of standard documentation was identified as a key constraint that the Bank might address. Skills are limited; few procurement specialists have experience in this area. The Bank could also make better use of dialogue during the selection process to arrive at a PPP concession agreement, for example, as the European Union allows. Countries following European Union practices might employ “competitive dialogue,” where a dialogue is conducted with bidders throughout the procurement process, aimed at obtaining the best cost and technical terms for the PPP agreement. The Bank can also make efforts to accept certain commercial considerations, such as the unwillingness of larger players to share all information with a minor player, if the information is considered commercially sensitive. This reportedly happened on one Bank PPP investments (India). When a concession is awarded on the basis of competitive bidding, the guidelines allow procurement to follow the procedures of the concessionaire. Yet conflicts of interest can still arise, for example, when a concessionaire awards over-valued contracts to an affiliate. It is conceivable that competitively selected concessionaires enter into conflicted downstream agreements, whereas those not competitively 84 CHAPTER 2 ADAPTING TO EVOLVING NEEDS selected pursue best value procurement. The Bank might consider other methods to control conflict of interest, such as through due diligence of the concession arrangements, as undertaken by private parties. It seems unlikely that the Bank can take a purely private sector approach to PPP, yet, somewhat like IFC, the Bank too needs to be willing to adapt to the legal framework of the project country, in part to avoid putting other parties in a situation of conflicting requirements. One complication for the Bank is that IFC might act as financial advisor. With its stronger orientation toward the private sector, it may not provide procurement-related advice that meets Bank policy. To the extent that IFC and the Bank are seen from the outside as the same agency, this can confuse clients. PPP contracts are output based and of long duration. They are designed to deliver a service (transport services, for example) and might persist for periods in the range of 20 years. This makes them quite different from a project to provide inputs (to construct an airport, for example) that only last long enough to complete construction. Over the duration of a PPP contract, assumptions of outputs (volume, pricing) and economic conditions may need revising. To manage PPP project outputs, contract management becomes as important as the process by which the contract is arrived at (procurement), and guidance on good practice in contract oversight may be needed. 85 3. Managing Risk and Increasing Transparency The Bank has a highly developed framework to manage procurement risk (Box 3.1). 88 Ex ante controls are present at the transactional level with procurement staff involvement in the prior review of a large proportion of individual transactions. Ex ante controls also operate through the assessment of procurement risks at the overall project level. The Bank also employs ex post controls on procurement risk, on samples of post-reviewed contracts. There are mechanisms for complaints and allegations of wrongdoing in procurement, and in select cases for investigations of allegations by its Integrity Vice Presidency (INT). Box 3.1. Procurement Risk Management in Recent IEG Evaluations IEG’s Review of IDA Internal Controls: An Evaluation of Management's Assessment and the IAD Review (2009c) found that there were generic weaknesses in controls over financial management and procurement processes at the Bank, in part because of regional variations in process. IEG found that no controls explicitly addressed the risks of fraud and corruption in IDA operations. IEG’s subsequent evaluation of the Bank’s remediation program (IEG 2010d) acknowledged the strengthening of fiduciary controls through the introduction of new risk-management tools but pointed out that the new systems would take time to be effective. The World Bank Country Level Engagement on Governance and Anticorruption (IEG 2012d) pointed to a lack of consistent and systematic treatment of corruption risks related to procurement. It cited the focus on ring-fencing methods in some countries but not in others as a reflection of the Bank’s inconsistency in setting risk tolerances. The evaluation also reported that the initiatives designed to manage the Bank’s reputational risks relating to governance and anticorruption were not necessarily the same as those that would help countries take on calculated development risks. Sources: World Bank 2011g; IEG 2009c, 2010d. These systems were reinforced in recent years following increased focus on fraud and corruption and reports of deficiencies in procurement risk management. This chapter reviews present procurement risk-control measures to inform the Bank’s proposals to move toward procurement processes that may require new trade-offs between risk and efficiency. IEG’s evaluation complements a recent review of procurement risk undertaken by the Bank’s Internal Audit Vice Presidency (IAD). Whereas IAD examined the level of compliance with project-level procurement risk-assessment tools, IEG reviews qualitative aspects of their performance in terms of preventing procurement risk. 89 This analysis 86 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY focused exclusively on process (or control) risk, because there are no indicators or tracked data on outcome risk. The Bank’s norms and directions in terms of procurement risk, and fighting fraud and corruption, need to be cognizant of international trends in this regard. Governance in public procurement is increasingly on the agenda of international organizations, underlining the need of countries to adjust to international norms if they expect to be part of the international economy. 90 Main Findings  Viewing all aspects of present procurement risk management, the overall conclusion is fairly positive, albeit qualified. In terms of design, the Bank has a highly articulated, and recently strengthened, set of risk-management instruments. However, they could be sharpened in focus, better integrated, and better used, in terms of data input and analysis of findings. The balance between specific tools could be revisited, in the light of a risk-efficiency trade-off.  In terms of results, most current measures of risk failure appear to be within acceptable levels of risk tolerance, based on numbers of misprocurements declared each year; procurement complaints, relative to bid awards; and INT referrals in relation to perceived fraud and corruption indicators in the procurement process. Ratios fall in ranges below 1–2 percent of contracts by number and value. Cases where the evidence from Post-Procurement Reviews/Independent Procurement Reviews (IPRs) shows that some countries to be seriously at odds with Bank procedures are already being monitored and managed.  The Bank puts considerable emphasis, in terms of its present risk-management framework, on ex ante risk controls through mechanisms such as prior review. Prior review instruments only partially reflect country or project risk. Overall, they could be relied on less as a risk control mechanism.  The Bank effectively applies its most restrictive and thus slowest review process (prior review of ICB contracts) to its most competitive and therefore least risky procurement method. 91 There is likely to be scope to reduce the risk efficiency trade- off by reducing the share of contracts that are prior reviewed and focusing prior review on high-risk contracts, as opposed to value based thresholds. “Methods” thresholds could incorporate better use of market information.  The findings of Post-Procurement Reviews and IPRs could be more strategically used.  Procurement risk was already being managed to a generally high standard prior to the introduction of the Procurement Risk Assessment Management System (P- RAMS). It does not appear that the correlation between risk identification and risk mitigation has been much improved, although P-RAMS is a well-intended effort to 87 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY offer more focused and standardized assessment and mitigation of procurement risks.  In principle, a key positive feature of P-RAMS is its dynamic aspect, now beginning to be articulated. Yet excessive frequent sequencing may be counterproductive. However, the P-RAMS template and process can be cumbersome and time consuming, especially with multiple implementing agencies, and it may foster a rigid “check the box” approach that limits added value. Risks identified in P-RAMS are not weighted or prioritized and can lead to index aggregation errors. Although the P- RAMS system has a facility for procurement staff override of automatic weightings, there is little evidence that this is exercised. Finally, the P-RAMS template could further sharpen its focus on fraud and corruption by featuring that risk as an additional risk factor.  Fraud and corruption risk is systemic and more Bank focus on the overall country level, rather than the transactions level, could increase effectiveness.  Client capacity issues remain a key risk area. Introduction Procurement risk in the Bank is not clearly defined or clearly distinguished from overall project risk (Box 3.2). Moreover, Bank focus, in terms of risk, tends to be on process risk (that is, non-compliance), rather than on outcome risk (that is, the risk of certain procurement objectives, or wider development objectives, not being achieved). IEG’s analysis necessarily also focuses on process risk, in the absence of instruments for or measures of the latter. The analysis focuses on the Bank’s principal present instruments to manage and to measure procurement risk, beginning with its core system of procurement thresholds, principally for prior review; its post-review system, and its risk assessment systems, notably the relatively recent Procurement Risk Assessment Measurement (P- RAMS) system. IEG also reviews available information on a series of measures of risk failure (misprocurement, complaints, fraud and corruption indicators) and the integration of procurement risk management with other areas of Bank risk management. Managing Procurement Risk—Prior Reviews and Thresholds CONCEPT AND USE OF THRESHOLDS A central element in the Bank’s ex ante risk management framework for procurement has been the establishment of an interlocking system of thresholds, defined in terms of contract value, which determine the level of scrutiny a procurement contract receives from Bank staff as well as the method of contracting applied (Table 3.1). Thresholds are critical tools for risk management, as they represent key pivot points in the trade-off 88 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY between procurement risk and efficiency. Thresholds are intended to reflect perceptions of risk to which the contract may be subject, in terms of the broad achievement of the Bank’s fiduciary objectives. Other factors reflected are conditions in the client country, in terms of country-level procurement capacity, the state of development of the private sector, and the quality of governance. At the project level, Bank procurement thresholds also take into account capacity of implementing agencies, based on their prior experience and record of performance, assessed in a procurement capacity assessment. Box 3.2. The Bank’s Taxonomy of Procurement Risk Bank procurement guidelines (OP/BP 11.00) do not define procurement risk. Some Bank reports refer to project risk, which has many nonprocurement components as “inherent risk,” whereas procurement risk is called “control risk.” In its turn the Bank’s procurement risk assessment measuring system, the P-RAMS template (which deals exclusively or mainly with procurement risk), calls its main risk element “project risk.” The Bank could benefit from more clearly defining a risk hierarchy, differentiating between types of procurement risk and distinguishing procurement risk from general project risks. Benefit would also be obtained from a unified risk taxonomy, so that all risk instruments whether in procurement or elsewhere, use a common terminology. The P-RAMS template should be amended to refer to procurement risk as “assessed risk” (premitigation) and “residual risk” (after mitigation). “Project risk” could be retained, but should be defined to include all inherent risk factors, including non-procurement factors. Source: IEG. Thresholds for prior review occupy the most important role in risk management, as they determine contract sizes, above specific values, that require the Bank to intervene upstream in the procurement process. Although the implementation of procurement in the Bank system rests with the borrower, the Bank specifically provides its no-objection at specific stages (see Appendix C, section on “prior review” and “clearance” for further details). These include, in the broadest terms: no-objections to draft bid documents or requests for proposals, bid evaluation report and recommendation for award, and any contract amendments. No-objection stages depend on the type of items procured (usually referred to as the category)—goods, works and consultant services. Prior review thresholds have been defined at a Bank-wide level, to set global maxima, and below this, at a country level, determined by regional procurement managers. These in turn are intended to serve as maxima at the level of individual projects, where project thresholds are determined by the task team leader and procurement specialist concerned, based on procurement risk assessments and the assessed capacity of the implementing agency as set forth in the procurement plan (World Bank 2011f). 92 89 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Table 3.1. Prior Review, Procurement Method and Clearance Thresholds Type of threshold (revisions) Description Threshold set by Prior Review Threshold Level at which all contracts with an There are three different prior review thresholds: (1997, 2008, 2009) aggregate value above the threshold must Bank-wide prior review thresholds set by be prior reviewed by Bank procurement staff OPSOR; regional prior review thresholds set at and receive a “no objection” at pre the country level by the regional procurement established steps in the procurement managers; and project-level prior review process. Thresholds will vary by thresholds set by task team leaders and procurement category and method. procurement specialists taking into consideration the results of the procurement capacity assessment and assessed capacity of implementing agency. Procurement Method Threshold Level at which all contracts with an Method thresholds are set by the regional (Depends on Region) aggregate value above the threshold must procurement managers and they also can be set be procured through a specific procurement at the project level by the task team leaders and method. The most common threshold is for procurement specialists. NCB/ICB but there are also thresholds for Shopping and Short List National Consultants. For some countries the prior review thresholds set by the RPMs are the same as the procurement method thresholds. Clearance Level at which all contracts with an Thresholds/Mandatory Review aggregate value above the thresholds must Thresholds (1992, 2002, 2005, receive no objections by PS/PAS, RPM or OPSOR 2009) the OPRC. Source: IEG compilation. Note: ICB = international competitive bidding; NCB = national competitive bidding; OPRC = Operational Procurement Review Committee; OPSOR = Operations Risk Management, Operations Country Policy and Services Vice Presidency; PA = procurement assistant; PAS = procurement accredited staff; TTL = task team leader. In addition, prior review contracts are also subject to clearance thresholds, which determine who, in the Bank’s procurement hierarchy, gives the relevant no-objections to prior review contracts. At the lowest level, contracts are typically cleared by field office- based procurement staff or procurement accredited staff, and at one level up, by hub coordinators for a cluster of countries, especially in some regions such as Africa. Larger value contracts, usually above $25million but below $50 million, are cleared by regional procurement managers for each of the six regions. The largest contracts, above $50 million are subject to the clearance of the OPRC—under the office of the chief procurement officer, at the Bank’s center. 93 Related to prior review thresholds and sometimes coincident with them are methods thresholds that determine whether specific contracts will be undertaken through its ICB procedures; NCB; or less formal procurement systems, such as shopping. 94 Methods thresholds, however, depend not only on the risk perceptions of project or country entities, but also on the state of development of local markets, from the point of view of goods to be procured. The greater the extent to which local suppliers are able to meet a given need of the contracts, the higher the method threshold is likely to be. In some 90 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY regions and countries, at specific points in time, prior review and methods thresholds have coincided—thus contracts undertaken by the ICB method have been those also subject to prior review. However, in most countries, large NCB contracts can also be subject to prior review. Table 3.2 shows, within the group of contracts subject to prior review, that ICB contracts have declined sharply as a proportion of the number of contracts, and many contracts under other methods are also being prior reviewed, although ICB clearly dominates in terms of the total value of prior reviewed contracts. Table 3.2. ICB Prior Review Contracts for Goods and Civil Works (numbers and value) (FY00–12) Value of ICB No. of ICB No. of all prior ICB contracts No. of ICB Prior review contracts contracts as % of contracts as % of FY reviewed (G&CW) (G&CW) ($ (G&CW) (G&CW) ($ billions) prior review prior review contracts (000) billions) contracts (000) contracts contracts 2000 6.6 5.0 5.2 2.8 79 56 2001 6.5 4.6 5.2 2.5 81 55 2002 6.4 5.0 5.0 2.6 77 51 2003 6.5 4.8 4.8 2.4 74 50 2004 7.3 4.7 5.1 2.4 70 51 2005 8.0 4.5 6.0 2.1 76 47 2006 6.7 3.8 5.2 1.8 77 48 2007 8.4 4.1 6.7 2.0 81 51 2008 10.4 3.9 8.0 2.0 77 51 2009 9.6 4.6 7.1 2.0 75 43 2010 11.4 5.0 8.9 2.1 78 42 2011 12.3 4.7 9.5 2.0 77 42 2012 12.1 4.2 10.5 2.1 87 50 Total 112.1 58.8 87.3 28.9 78 49 Source: World Bank. Note: These data refer to goods and works alone. Adding consultant services reduces the proportion of ICB. Thresholds have been revised, though on an ad hoc basis, and systematic information on their evolution, and on previous levels of thresholds, is limited, partly because most such decisions have been taken at the regional procurement manager level on a country basis. 95 Looking at the past five years, Bank-wide prior review ceilings were found in 2008, set in terms of three risk categories (high, moderate, and low); revised shortly after in 2009 to a four-point risk scale (high, substantial, moderate, and low). The 2009 Bank-wide prior review thresholds are still in effect. Historical regional/country level prior review and method thresholds are available, in varying degrees of comprehensiveness, for each Bank region. 96 Clearance thresholds have been changed at least four times in the past decade. In 2002 the first Decision Authority Matrix that layed out clearance responsibilities for prior review as well as for single-source contracts, rebidding approval, and approval of request for extension of bid validity was used. A 2005 revision has Bank-wide thresholds for 91 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY clearance by regional procurement managers (RPMs) and by the OPRC, by procurement category. A further 2009 prior review revision outlines clearance thresholds for procurement assistants/procurement-accredited staff, regional procurement managers, and OPRC by procurement category (and by method for direct contracting and single- source contracts). 97 Although thresholds have been a central device for managing procurement risk and achieving transparency, they illustrate clearly the trade-off between risk management and other guiding principles: economy and efficiency. Lower prior review thresholds may reduce risk but are time intensive and increase procurement staff work load, possibly detracting from staff time spent on other aspects of procurement work, such as capacity development, as argued by the Bank’s procurement anchor. 98 Clients point to delays in the procurement process, especially when contracts have higher clearance thresholds and need Washington-based approval (as discussed in Chapter 4). Lower method thresholds may arguably increase potential participation of international bidders (thereby increasing competition) but could imply lost opportunities for local suppliers and markets, in addition to adding to time spent on procurement. In practice, depending on market capabilities, lower method thresholds may not even attract more international suppliers if there are a wide range of competent domestic producers (Box 3.3). PERCEPTIONS OF THRESHOLDS—IEG FIELD VISITS Review Thresholds and Clearance Thresholds Bank procurement staff acknowledge that current review procedures were perceived to be helpful for controlling fiduciary risk—reported by Bank procurement staff in Bangladesh. Private sector respondents (for example, in Azerbaijan) point out that reviews serve to limit mistakes overlooked by Project Implementation Units. Nevertheless, on balance, there was a view that current prior review threshold could be raised with beneficial impacts on workload and clearance time, and only limited impact on risk or competition. Recent increases in thresholds in many countries have had a positive reception among Bank procurement staff (Ethiopia, Mexico, Morocco, and Peru). And respondents from Indonesia, Azerbaijan, Bangladesh and Tanzania indicated that they should be raised. Respondents in Azerbaijan, Bankgladesh, the Philippines, Senegal, and Turkey also stressed the need to raise procurement method thresholds. In the Philippines, Bangladesh, Mexico, Turkey, and Azerbaijan, it was suggested by country respondents that insistence on ICB for procurement of goods and works as well as for the selection of consultants, when good domestic capacity exists, may produce 92 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY limited gain as ICB has a cost in terms of considerably longer process time required for no-objection. Insistence encourages local firms to take up joint ventures with foreign agencies where often the local partners beat the brunt of the burden of execution. Also in many parts of the Bank, ICB is synonymous with prior review; an approach questioned by Bank procurement staff who did not feel that large value necessarily implied high risk. Finally, there was a sense that risk aversion in the award of ICB contracts led to an over- emphasis on price, sometimes leading to questionable outcomes. 99 Calls were made for raising NCB thresholds accordingly (Azerbaijan and Tanzania) and also for making use of other methods. Box 3.3. Setting Method Thresholds to Optimize Competition at the AfDB Are there ways to raise methods thresholds, to enhance procurement efficiency, while minimizing the possible losses in terms of risk or competition? The AfDB shows that the trade-off can be quite accurately quantifiable, using relevant contract data. Using contract data from its accounting system, AfDB has shown that in works contracts foreign bidders entered bids on contracts for only those contracts in the top quintile (20 percent) of contract value. The AfDB concluded, accordingly, that to set the ceiling contract for ICB at any level lower than 80 percent of highest value would involve no increase in competitive bidding but would involve significantly more review work; put another way, if thresholds on ICB for works contracts were raised to around 80 percent of maximum contract value, there would be significant cost savings in processing time, but little if any loss of competitiveness, because foreign bidders would not in any case enter the market for the lower value contracts. The AfDB has conducted an analysis of all its methods thresholds on a similar conceptual basis which resulted in a new set of recommended thresholds for both works and goods, grouped into three sets of countries ranked by risk ratings. The World Bank has the capability to undertake analyses of methods thresholds, as illustrated above, even based on present Form 384 data. IEG’s analysis of the Bank’s methods thresholds also suggests that the bottom quintiles have a high share of local winners, although somewhat less, on average, than at AfDb (Appendix C). Overseas firms are also better represented in the top 5 percent. More disaggregated analyses would provide methods for a better quantification of the risk-efficiency trade-off. Systematic operational use could be made of such analyses, for example, for broadly comparable groups of countries and categories of goods and services Source: AfDB 2012. It was pointed out (Azerbaijan and Turkey) that there is value in increasing clearance thresholds so that the quantity of contracts that can be reviewed by procurement 93 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY specialists as opposed to RPMs would lower processing times. Bank task team leaders also subscribed to this view and pointed out that raising the country procurement office level of clearance would expedite the no-objection process for bidding awards. Responses also point to the dilemmas of revising thresholds. On the one hand, it was argued that with the present decentralized structure of the application of thresholds at the country level, there are potential differentials in the treatment of perceived risk across countries. One example offered by staff interviewed was that of Haiti, where all contracts are subject to prior review, compared to other arguably equally fragile states such as Burundi and Rwanda, where this has not been the case, although in the views of some procurement staff, country capacity levels may have been lower. Yet Bank procurement staff also perceive a need for more local flexibility in applying thresholds. In Indonesia, for instance, Bank procurement staff complained about the lack of flexibility for adjusting thresholds according to the capabilities of implementing agencies. Tanzania, by contrast, was pointed out as a clear example of threshold flexibility where task team leaders reported that the Bank raised prior review thresholds on two occasions to accommodate country circumstances and context. Procedures for handling complaints also appear to involve referrals to higher clearance levels. Field office procurement staff in some countries (Bangladesh) point out that referring even small, and sometimes technical, complaints to the regional procurement manager for clearance is time consuming and requires detailed prior preparation. Moreover, if not localized, there is inadequate contextualization and less understanding of the issues. Questions were raised about the value of the threshold mechanism for containing risk. Some procurement staff felt that implementing agencies see no objections as a requirement largely to meet audit objectives. In some instances, it was also felt that the review process made country clients overreliant on the Bank, diluting capacity development and giving clients a false sense of security. Several respondents pointed to the trade-off between risk containment and efficiency, showing a lack of clarity in the clearance process, especially because of clearance thresholds that restrict decision making by local procurement staff, and the absence of clear service standards at the Bank in terms of the response process. Country management perceived concern about fiduciary risk as substantially detracting from efficiency in project implementation. Though this was less of a concern to other stakeholders, there was a perception by all that ICB was used somewhat more than needed, from the point of view of achieving a balance between competition and the quality and efficiency of project execution. There were also concerns that although the Bank had good procedures for controlling procurement risk, this may not be enough for 94 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY the overall containment of fraud and corruption, much of which is thought to occur during implementation, that is, in the post-contract-award period (for example, bribery, short payments, contract variations, and so forth). More importantly, the Bank’s procurement processes, although ruling out the worst abuses (direct/noncompetitive contracting and the like), cannot overcome social and political systems where such practices may be commonplace. IEG ANALYSIS OF THRESHOLDS AND RISK MANAGEMENT As discussed in Chapter 4, feedback from country case study questionnaires suggested that current thresholds could be raised with beneficial impacts on workload and clearance time, with only limited impact on risk or competition. IEG undertook its own analysis of the application of procurement prior review and procurement method thresholds across Bank borrower countries, with the objective of seeing whether and to what extent current thresholds, especially for prior review, have been commensurate with risk exposure. IEG first measured the extent to which country-level prior review thresholds for goods and works correlated with select broad parameters of country governance /country fiduciary risk, including the Bank’s CPIA indicators, as well as external indices (Transparency International’s Corruption Perceptions Index and the Kauffmann and Kraay Governance Indicators) on the quality of governance. 100, 101 Second, IEG examined project-level thresholds in relation to project-level risk. Two samples were constructed, one from 2008 and one from 2011, for the comparison of Bank thresholds with external measures of country risk/ country governance. There are two caveats to the sample construction: the setting of thresholds is not standardized across regions; and the gaps in available information vary considerably across regions. Correlations between country-level prior review thresholds and governance indicators were undertaken in clusters, reflecting different practices in setting thresholds across regions. (Details are available in Appendix C, which describes data sources, sample construction, and correlation methods used in the presence of diverse threshold-setting and risk-rating practices across regions). RESULTS Regional Prior Review Thresholds—Correlations with Governance Indicators The overall result is that there is a weak statistical relationship between borrower country risk or country governance (as measured by the selected risk vectors) and Bank prior review risk thresholds. Table 3.3 reports results for all countries in aggregate; the main findings are summarized below. 102 • While the majority of the correlation coefficients are positive, most are moderate to weak in strength, at best around 0.5; there are also a substantial number of much lower correlations. 95 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY • Results are similar across different governance indicators, though somewhat higher for the overall CPIA, the CPIA for budget and financial management, and for the Kauffmann and Kraay indicator on regulatory quality. Results are the poorest for the Kauffmann and Kraay indicator on control of corruption, and also poor for the CPIA subindicator 16 on transparency, accountability, and corruption in public service. • There are somewhat lower correlations for NCB goods and works than for ICB goods and works. • Correlations in 2011 tend to be lower than those in 2008. These might suggest that inconsistent practices in setting thresholds are not improving over time. Table 3.3. Prior Review Procurement Thresholds: Correlations with Indices of Governance CPIA Sub- CPIA sub-indicator Transparency indicator 13. 16: transpar., International - Kauffmann Kauffmann quality of budget account. and Corruption and Kraay - and Kraay - CPIA - and financial corrup. in public Perceptions regulatory control of overall management sector Index quality corruption 2008 ICB goods (68) 0.51 (68) 0.45 (68) 0.32 (67) 0.46 (68) 0.57 (68) 0.25 2008 ICB works (68) 0.46 (68) 0.41 (68) 0.18 (67) 0.30 (68) 0.43 (68) 0.16 2011 ICB goods (117) 0.33 (117) 0.32 (117) 0.20 (123) 0.12 (123) 0.28 (123) 0.15 2011 ICB works (116) 0.41 (116) 0.36 (116) 0.24 (120) 0.24 (122) 0.40 (122) 0.20 2008 NCB goods (51) 0.39 (51) 0.39 (51) 0.25 (51) 0.27 (48) 0.45 (48) 0.14 2008 NCB works (54) 0.39 (54) 0.36 (54) 0.00 (54) 0.11 (54) 0.35 (54) -0.04 2011 NCB goods (32) 0.29 (32) 0.09 (32) 0.26 (37) 0.14 (37) 0.44 (37) 0.29 2011 NCB works (33) 0.38 (33) 0.26 (33) 0.27 (38) 0.21 (38) 0.51 (38) 0.35 Source: IEG analysis. Note: Numbers in parentheses indicate numbers of observations. CPIA = Country Policy and Institutional Assessment; NCB = national competititve bidding. Although these findings are subject to caveats on interpretation, because of some regional data being unavailable and certain difficulties in comparison across regions, they suggest that at best it is difficult to reflect country procurement risk (as reflected in indices of governance and corruption), in the present system of value based indices, with consistency across the Bank. Project-Level Prior Review Thresholds It can be argued that country-level risk thresholds are not the ones that actually apply to individual projects; instead, project level thresholds are used, based on risk specific to the individual implementing agency. IEG therefore also analyzed project level procurement thresholds, and their association with project-level risk (Appendix C). An initial review of prior review thresholds at the project level demonstrates that although project-level prior review thresholds are usually set at the level of the regional prior review threshold (some 50–60 percent in the sample below), there are some 96 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY exceptions, especially projects with thresholds set below the regional/country threshold (15–30 percent) and perhaps more interestingly, a few (20–25 percent) projects that have prior review thresholds set above the country threshold (Figure 3.1). The Europe and Central Asia and East Asia and Pacific Regions have had more project-level flexibility, with projects below or above the country prior review thresholds, compared to greater uniformity in South Asia and Africa. Figure 3.1. Country and Project-Level Prior Review Thresholds by Region—Civil Works and Goods Source: IEG analysis. Note: For civil works n = 65, for goods n = 67. Regions: AFR = Africa; EAP = East Asia and Pacific;ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. Next, data were collected on the residual procurement risk rating at the level of each project (Figure 3.2). This information was associated with information on project level risk thresholds, to see if there was a relation between, for example, relatively lower project risk levels (compared to country risk) and relatively higher project risk. Figure 3.2. Procurement Risk Ratings for Projects Above and Below Country-Level Prior Review Thresholds Source: IEG analysis. Note: For projects with project threshold > country threshold n = 23, for projects with project threshold < country threshold n = 26. 97 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Contrary to assumptions about project risk levels and relative thresholds, projects with thresholds above country-level thresholds tended to have somewhat higher levels of procurement risk (more projects in the substantial risk category) than projects with thresholds set below the country prior review threshold. The finding suggests again that in aggregate, the application of project-level prior review thresholds has a blurred association with procurement risk. CONCLUSIONS AND IMPLICATIONS The Bank’s prior review thresholds, as presently defined in value terms, do not seem to have a consistent or strong relationship with governance or risk, whether at the country or the project level, although there is some weak positive association. How can these findings be explained? One factor is the likely poor association between value thresholds and risk levels. Large contracts may not necessarily be more risky, and setting thresholds in value terms may capture statistical noise. Other reasons could be the use of different implicit scales in different regions or countries that are not due to differences in risk. There may be a tendency, for example, to “scale” thresholds to loan size or country income, rather than country risk alone. One implication of these findings is that setting an ever-finer grid of value-based thresholds is not likely to significantly improve the management of risk. Some simple broadbanding may be adequate, to the extent that value-based thresholds are to be maintained. A second implication is that differentials in treatment of risk across regions suggest that a uniform Bank-wide system, based on transparent criteria, would be useful. Third, consideration could be given to moving away from value-based thresholds and focusing instead on risk-based criteria. Such a risk-based focus would have least scope for error if determined at the level of individual contracts, instead of broad categories such as goods or works. Nevertheless, the Bank should be cognizant of the increased potential for complaints of arbitrariness in judgments that may be leveled as a result. Managing Procurement Risk—Post-Procurement Review Post-Procurement Reviews assess whether procurement conducted by the client’s implementing agencies, without Bank involvement, has been executed in compliance with Bank procedures. More rapidly processed, but riskier than prior reviews, procurement post reviews are conducted on an agreed sample of all contracts designated for post-review. Designations are made by project staff, agreed with the borrower, and then written into the procurement plan. Most often the sample comprises 20 percent of the total number of designated contracts, but this varies by project, depending on procurement specialist views of project risk. Project staff often use the sample size for post-review as an instrument of risk management (increasing the sample for riskier 98 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY projects) and may also require more frequent than the usual annual postreview cycle. In the sample of 69 project appraisal documents (PADs) that were reviewed covering the period 2002–10, 24 projects explicitly declared a post-review sample below). Of these, 15 required a 20 percent sample but many others varied between 5 and 40 percent. Two projects asked for a 100 percent review. IEG reviewed both completion rates of Post-Procurement Reviews and their findings (in terms of procurement malpractice or suboptimal outcomes). 103 The Bank’s annual procurement reports give considerable emphasis to Post-Procurement Review completions by the regions, and compliance has generally improved in recent years: The FY11 report shows that overall the Bank was 99 percent compliant (that is, the number of actual postreviews, compared to the designated number subject to postreview), with three regions (Latin America and the Caribbean, the Middle East and North Africa, and South Asia) being 100 percent compliant; Africa, East Asia and Pacific, and Europe and Central Asia were 97 percent compliant. 104 Table 3.4. Share of Contracts Designated for Post Procurement Review (2002–10) Percent review required 5 15 20 30 40 100 Numbers of projects (total: 24) 1 2 15 3 1 2 Source: IEG analysis. The format and content of the Post-Procurement Review contains information that is operationally significant: it includes an assessment of procurement procedures employed; compares these to the loan agreement; sees to contract management issues (assesses technical compliance and physical completion of the contract); assesses the capacity of the implementing agency and its systems; identifies noncompliances in executed procurement; and recommends remedies. The format also includes a requirement to complete two key risk ratings, going forward: the assessed risk for the procurement process and the risk expected for implementation of contract management. In some respects (for example, contract management) this is a more rigorous agenda than is contained in the prior-review process, and it has the advantage of occurring in real time (as against ex ante, before project start up), so it benefits from an element of actual implementation. To this extent the Post-Procurement Review is a useful monitoring tool to ensure the integrity of the Bank’s procurement processes, and it appears to be used as such within some regions. The Middle East and North Africa Region, for example, developed consolidated follow-up action plans, to remedy identified deficiencies. The Post-Procurement Review system was recently improved by enabling electronic uploading of its content (including the risk ratings) into the Operations Portal, to link to both P-RAMS and the ISR. It is not clear whether these linkages are fully operational yet, but it is clear that they are a key subject in the integration project that is being worked on by the Bank’s procurement anchor intended for full operation by end of FY14. 105 99 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY The Bank’s annual procurement reports, as regards Post-Procurement Reviews, focus more on compliance than operational content. 106 This limited treatment underrates the potential importance of Post-Procurement Reviews as a risk-management tool, and its potential for directly checking and validating (or discounting) whether procurement performance under postreviewed projects is of a higher or lower standard than that under prior-reviewed contracts, despite ample data being collected at the regional level. Part of the reason is that most regions do not track and analyze consolidated Post- Procurement Review data. From two studies that did so (both in South Asia: Bangladesh and India), there was information that showed that (for India) of those contracts post- reviewed, 24 percent by number and 32 percent by value of contracts were not in compliance with Bank procurement guidelines or the loan agreement. In Bangladesh, findings were similar: 33 percent of contracts by number and 26 percent by value were found to be subject to “major observations.” 107 In the Middle East and North Africa Region a summary report on Post-Procurement Reviews/IPRs undertaken for FY09 and FY10 shows major and concerning deviation from Bank procedures in some countries (for example, the Republic of Yemen), whereas in other countries (Morocco) noncompliance is of a more routine and relatively minor nature. In yet other countries (Turkey, Ukraine) Post-Procurement Reviews in FY11–12 revealed that post-reviewed procurement practices were entirely exemplary. Appendix Tables C.5 and C.6 contain a listing of the Post-Procurement Review reports consulted for this review and a summary of exceptional irregularities found in the case of the Yemen Post-Procurement Review. IEG conducted an analysis of mandatory risk ratings in Post-Procurement Reviews. Procurement staff have to rate the expected future risks of procurement, as well as, interestingly, contract administration. Data from Post-Procurement Reviews suggest only moderate risk ratings going forward (Figure 3.3). Figure 3.3. Distribution of Risk Ratings in Post-Reviewed Contracts Source: IEG analysis of select PPRs. 100 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY IEG also analyzed the extent to which risk ratings under post-reviewed contracts were higher or lower than procurement risk ratings in the P-RAMS database. Figure 3.4 compares evidence on procurement risk ratings from two sources: first, from 520 projects contained in the Bank’s Post-Procurement Review database, which show risk ratings for procurement process and contract administration; second from a general sample of 542 projects for which 668 P-RAMS templates were completed during June and July 2010. The distribution of risk ratings is significantly different. In the Post-Procurement Review risk ratings for procurement process 345 of 520 ratings (66 percent) were rated moderate or low risk; and, similarly, for contract administration 349 out of 520 ratings (67 percent) were in these lower-risk categories. By contrast, in the general project sample, procurement risk was rated high or substantial in 402 of the 668 projects in the sample (60 percent of the total). Figure 3.4. Comparison of Project Procurement Risk (from P-RAMS) and PPR-Based Risk Ratings Source: IEG analysis. Note:Bbased on a sample of 668 completed P-RAMS templates for 542 projects in the period July 2010 to June 30 2012; and review of a sample of PPRs. PPR = Post-Procurement Review; P-RAMS = Procurement Risk Assessment Management System. A first caveat is that the two data sets are not comparable, because the Post-Procurement Review sample is essentially contract-based from postreviewed projects only, and the general sample refers to project risk, and contains both prior- and postreviewed contracts. A second caveat may be that P-RAMS-based risk assessments are made ex ante whereas the Post-Procurement Review assessments are made during implementation, with the benefit of at least some track record of how implementing agencies have been performing. Taken overall, therefore, the evidence emerging from the findings of Post-Procurement Reviews offers mixed results: at least in some consolidated country reports, Post- Procurement Reviews have shown that post reviewed contracts have been noncompliant with Bank procedures in some 24–30 percent of the sample, with “serious observations” in other cases. By contrast, a much larger sample of Post-Procurement Review-based risk 101 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY ratings suggests that procurement specialist staff perceive expected risks of failures (in procurement or contract administration) to be distinctly lower than in a general sample of all projects. As with the analysis of thresholds, it suggests that a more detailed Bank-wide review of its risk instruments and their correlations would be useful and might be a basis to revisit current country risk ratings. Finally, mention is made of the IPR, an elective procurement review tool that regional RPMs can use to review procurement performance in a single project or batch of projects, where there may be specific concerns. IPRs are often done to supplement Post- Procurement Reviews; their results are recorded at the regional level and are used for follow-up by the RPM and the procurement staff. IPRs review the performance of both the Bank and the borrower and have been used to provide feedback and guidance on procurement risk-management issues to task teams and implementing agencies alike. In FY11 there were 54 IPRs completed in 14 countries covering 54 projects and 725 contracts. These have not been analyzed by IEG. Managing Procurement Risk—The P-RAMS Instrument MANAGING RISK BEFORE P-RAMS The Procurement Policy Framework The identification of potential procurement-related risk has been recently formalized in the Bank through P-RAMS, with 11 specific risk indicators (Box 3.4) (World Bank 2010f). A recent review by the Bank’s Internal Audit Department examined levels of completion of the P-RAMS instrument. IEG’s analysis seeks to undertake a qualitative review of the extent to which the new instrument has actually led to an improvement in the Bank’s ability to measure and track procurement risk and assess implications for overall project risk. This analysis therefore undertakes a comparison of risk measurement in the period before and after the introduction of the P-RAMS instrument. IEG also reviews evidence from country case studies on the design, user friendliness, and perceived value added of the instrument. 102 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Box 3.4. The Procurement Risk Assessment and Management System The P-RAMS Users Guide segregates risks into two types: inherent risks and control risks. Risks are measured starting with the overall country and institutional setting (strategic and operational risks that most projects have to take as given), moving to the agency level (where there is scope for the project team in support of their country counterparts to influence the level of risk), and at the project procurement level (where there is the most scope for affecting fiduciary risk levels through establishing compliance and reporting measures). P-RAMS was introduced in response to the IDA Controls review, which noted deficiencies in the management of procurement risk. Its integration into the Bank’s overall risk management framework is envisaged. IEG’s evaluation of the progress made with remedial measures acknowledged the strengthening of fiduciary controls through the introduction of P-RAMS, but pointed out that the new system would take time to be effective. The 11 risk factors contained in the P-RAMS template: 1: Accountability for Procurement Decisions in the Implementing Agency 2: Internal Manuals and Clarity of the PR process 3: Record Keeping and Document Management Systems 4: Staffing 5: Procurement Planning 6: Bidding Documents, (pre-) qualification, short-listing, and Evaluation Criteria. 7: Advertisement, Pre-bid/Proposal Conference and Bid/Proposal Submission 8: Evaluation and Award of Contract 9:Review of Procurement Decisions and Resolution of Complaints 10: Contract Management and Administration 11: Procurement Oversight. Aggregated Risk Measures In the P-RAMS template (completed by the risk assessors on line) each risk category contains several individual questions (as few as three or as many as 10) for a total of 60 individual questions. If the assessor identifies other risks not contained among these 60 questions, he/she can add these new risks by clicking on a “customize risk” facility. A rating is ascribed to each risk factor and these ratings are aggregated by the template to give a calculated risk rating for each risk category. If the assessor disagrees with the overall calculated rating, he/she can override the rating and insert a new one, but this requires an explanation in a text box provided. So, although the template provides implicit equal weighting to all 11 risk factors, the assessor can in principle change this by inserting different weightings. The template also contains a Risk Assessment Summary section, including proposed mitigation measures. Sources: World Bank 2010b, 2012i. 103 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Approach and Method The empirical basis for the analysis was a sample of PADs. 108 IEG reviewed the sections that dealt with project and procurement risks and, making a qualitative assessment as to their content and quality, focused around two key components of the procurement annex and the procurement plan: the project’s implementing agency capacity assessment (herein referred to as “risk assessment”) and the procurement risk-mitigation plan (herein referred to as a “mitigation plan”), which contains the specific measures to be taken to address and manage the risks found in the risk assessment. 109 The assessment compared risk assessment and management in two groups of projects: before and after introduction of P-RAMS. The results of these groups were then compared. The analysis on the pre-P-RAMS sample was conducted mainly on PADs; the analysis after P-RAMS was introduced was conducted mainly on the content of the P- RAMS template. The pre-P-RAMS projects were approved during the years 2002–10; those with P-RAMS were for projects approved after July19, 2010, the date when P-RAMS became mandatory for all projects at the project concept note stage. The assessment began with the pre-P-RAMS group and focused specifically on the capacity assessment and risk mitigation plans in each project, but also collected data on other related aspects, including country context, links between procurement risk and overall project risk, and use of thresholds. A scorecard was constructed for the risk assessments, consisting of risk categories most commonly found in the PADs. A similar scorecard was constructed to score mitigation measures in the mitigation plan for each project. In theory, P-RAMS also provides for “custom” risks to be identified (that is, those not contained in the specified 11 risk factors), but in practice (judging from the sample) it does not appear that this feature was much used. Analysis and Findings • Procurement risks have generally been well identified, and this has improved in recent years, even before the introduction of P-RAMS, although there are some deviations from this norm. This assessment was made based on a review of: the number of risk categories addressed, the number of specific risks identified and assessed, the linkage of procurement and project risk, and the number of items contained in the procurement risk-mitigation plans. The average number of risks addressed per project was about 12; seven projects addressed more than 20 risks (23 was the highest number); 14 PADs addressed fewer than 10 risks, of which 2 (3 in Latin America and the Caribbean and 1 in Europe and Central Asia) addressed no specific risks (Table 3.5). 110 104 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Table 3.5. Risk Categories and Individual Risk Identification in Pre-P-RAMS PADs by Period 2002–06 2007–10 2002–10 Number of risk categories 4.7 7.2 6.3 Number of individual risks 8.3 14.9 12.3 Fraud and corruption risk Items 0.7 1.9 1.4 Source: IEG calculations from scorecard from a sample of 39 pre-P-RAMS PADs. Note: Average Incidence per PAD. PAD = project appraisal document; P-RAMS = Procurement Risk-Assessment Management System. • Implementing agency capacity is the dominant risk category, both before and after P-RAMS. However, the single most frequent risk category before P-RAMS was the catch-all category “other,” reflecting individual project differences in risk profiles, but also nonspecification in the template. 111 • Elements of fraud and corruption risk were frequently identified in the project sample (Table 3.5 and Figure 3.5). Fraud and corruption risk was the third most frequent category of risks identified. 112 • In terms of the mitigation plans, the most frequent remedy proposed was the appointment of consultants and training of implementing agency staff. Of the 39 sampled projects, 31 contained measures to employ consultants in the project implementing unit and 30 involved training of local procurement staff. Nineteen projects were implemented by project implementing/management unit. • Measures to ensure transparency were also often featured in the mitigation plans (for example, to publicize awards, open or use project-related websites, and use of civil society in oversight). • Comparing infrastructure (higher value) and social sector (typically lower value) projects, infrastructure projects contained a higher share of lower-risk projects (Figure 3.5). This appears to illustrate that large value may not reflect high project risk. Another possibility is that greater use of more competitive procurement methods that use more prior review (ICB contracts are more common in infrastructure than in social sectors) may imply that such projects are viewed as less risky. • PADs make few explicit cross references to Country Procurement Assessment Reports (CPARs) when discussing risk or capacity issues. Explicit reference to CPARs completed in given countries was mentioned in a limited number of cases in the project sample, although they often are sources to highlight risk factors in local implementation of project procurements. From the sample of 39 pre-P-RAMS projects (2002–09), only 15 mentioned the CPAR. In the 24 post-P-RAMS projects (2010–12), 5 mentioned CPARs. 113 105 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Figure 3.5. Procurement Risk Categories, Risk Mitigation, and Risk by Sector—A Sample Source: Project PADs from 39 projects in 20 countries approved before P-RAMS (2002–10). Note: Only 17 of the 19 infrastructure projects and 16 of the 20 social sector projects in the pre P-RAMS sample had initial procurement risk ratings – these were used for the second set of graphs. Integrating Procurement and Project Risk: Although procurement performance is closely associated with overall success in project implementation, most projects give limited regard to the impact of procurement risk on overall project risk (see Chapter 1). Evidence of this was found by tracking how frequently procurement risk issues were featured in the Critical Risks Table of the PAD: in total, the 39 pre-P-RAMS projects in the sample listed 378 critical risk items, of which only 58 (about 15 percent) were related to procurement. However, there were also projects in which procurement risks were prominently featured, together with thoughtful risk-mitigation measures (Box 3.5). A key purpose of the proposed link between P-RAMS and the Operational Risk Assessment Framework (ORAF) is to provide an automatic platform to display procurement risk as part of overall project risk, but this has yet to become universal practice. 106 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Box 3.5. Integrating Procurement and Project Risk—Tanzania Second Transport Corridor Project The PAD for this project contained a highly articulated and clearly documented account of how procurement risks could impact the overall success of the project and how these risks were closely integrated into overall project risk, as follows: it had a complete, itemized listing of initial risk ratings, descriptions of specific mitigation measures, and corresponding residual risk ratings. It used project covenants to ensure hiring of procurement specialists; it used advanced procurement to avoid delays; it required post- qualification to ensure quality; and had penalties for operators who failed to deliver service (that is, contract management clauses). Of 25 critical project risk factors, 13 were procurement related, of which 8 were rated high or substantial and none low. Mitigation of procurement risks (10 had residual risk rated as moderate or low; none was rated high) contributed significantly to mitigating overall project risk. High Substantial Moderate Low Project risk Initial rating 10 5 8 2 residual rating 7 6 12 Procurement risk initial rating 5 3 5 residual rating 3 4 6 Source: Tanzania Second Central Transport Corridor project, April 2008. To conclude, prior to P-RAMS, the treatment of procurement risk assessment and risk- mitigation measures was generally of good quality, but also with a considerable degree of variability. The treatment of procurement was only loosely standardized in the PADs. PADs generally captured core risk items such as lack of capacity among implementing agency staff, lack of experience with Bank procedures, or poor record keeping, but the data also show a wide dispersion of “other” risk items. There was a discernible increase in the focus on fraud and corruption risk as part of procurement after 2007. THE INTRODUCTION AND CONTRIBUTION OF P-RAMS Intended Contributions—Sharper Risk Focus and Dynamic Aspects Pre-P-RAMS measures addressed procurement risk in project PADs, that is, mostly upstream during project design and preparation. Even though Procurement Plans (in principle, updated annually) and ISRs were intended to track progress with procurement as part of project implementation, a need was expressed for a more downstream and dynamic risk management tool. The response was P-RAMS, mandated for all projects with a Project Concept Note written on or after July 19, 2010. P-RAMS was not intended to either replace existing practices for managing procurement risk or necessarily add new information; the principal purposes were to sharpen the focus 107 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY of risk assessment and the content of the mitigation plan and to codify and standardize the treatment of both aspects. Under P-RAMS, the risk assessment has been replaced by the Procurement Risk Assessment Questionnaire, and the mitigation plan has been replaced by a Mitigation Measure Action Plan. 114 Comparing this with the risk-assessment questionnaire, residual risk ratings are ascribed, based on the perceived likely effectiveness of the mitigation measures. An overall residual risk rating is thus assessed for each implementing agency and thence for the project as a whole. In principle, P-RAMS provides a way to monitor risks and remedial actions over time. Where projects may have more than one implementing agency, a separate Procurement Risk Assessment Questionnaire and Mitigation Measure Action Plan is conducted for each. 115 After the introduction of P-RAMS, as part of the Bank’s efforts at Investment Lending reform towards a more risk-based approach—an overall ORAF was designed, to integrate all key risk dimensions relating to Bank projects into a single online platform in the Operations Portal. It was a logical step to link P-RAMS to ORAF, as a means of including procurement risk as one of the prominent risk elements. The online link systems are mostly in place. ORAF includes fiduciary information (in indicator #3 “Implementing Agency Risk,” which also includes fiduciary risk), though input of this information is not mandatory. Although there is a link to “Procurement Risks” as a tab in ORAF (only for risks rated high or substantial), staff often are not aware of its existence and represent that there is little reflection of procurement risk in the overall project risk management framework. 116 P-RAMS risk ratings are intended to be conducted in several sequences throughout a project’s life, and also to be integrated into project ISRs. There is also a link in the P- RAMS template intended to be informed by outcomes of the periodic Post-Procurement Reviews of contracts that fall into the sample for a given project, so that these various tools form a dynamic monitoring chain that can provide an integrated system of risk monitoring. The dynamic feature of P-RAMS has been stated to be its central attribute. Implementation and Sequencing of P-RAMS Despite its mandatory nature (required for all investment loan projects in the Project Concept Note phase on or after July 19, 2010), P-RAMS was not adopted universally across all regions and it has taken some time to build compliance across the Bank. At the end of FY12 the number of projects for which at least one P-RAMS template had been completed was 542 of 794 projects that were eligible for P-RAMS, a compliance rate of 68 percent. 117 These less than complete compliance rates were built up after a slow start, and reflect a major “compliance push” in many regions before the end of FY12 (Figure 3.6). P- 108 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY RAMS remains a new tool, and staff are not fully familiar with its attributes. Table 3.6 shows compliance rates by region. Figure 3.6. P-RAMS Completed by Month (2009–12) Source: IEG calculations from P-RAMS database provided by OPSOR. In terms of its dynamic contribution, IEG analysis shows that 90 projects of the 542 had completed P-RAMS by June 30, 2012, with more than one sequence and 25 projects that had more than two sequences. This suggests that it will take time for the dynamic aspect to be more fully developed. A question may also be raised as to the required frequency of repeated P-RAMS sequences. The spacing between P-RAMS sequences appears quite short: in some observed cases, only three months. It appears unlikely that risk factors would change so rapidly and it also brings into focus whether procurement efficiency is well served by such frequent sequencing. The dynamic attribute of P-RAMS may need to be more carefully articulated. Table 3.6. P-RAMS Compliance Rates by Region AFR EAP ECA LCR MNA SAR TOTAL Eligible projects 255 147 108 87 72 125 794 Percent completed 58 80 81 62 61 72 68 Number of projects with more than one P-RAMS sequence completed Sequences: Two 12 14 19 7 3 10 65 Three 4 1 3 2 2 2 14 Four/Five 0 3 1 1 0 5 11 Source: OPSOR P-RAMS database. Note: Compliance is defined as at least one P-RAMS sequence being completed. AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. 109 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY PERCEPTIONS OF P-RAMS AND ORAF: IEG FIELD VISITS Bank staff and country clients gave mixed reviews to new risk management tools, such as P- RAMS, and their relation to a wider project risk management framework--ORAF (Box 3.6). 118 EMPIRICAL ANALYSIS OF P-RAMS DATA Comparison of P-Rams and Pre-P-Rams Risk Information IEG compared data gathered for the pre-P-RAMS control group (Figure 3.5) with a sample of projects using P-RAMS (Figure 3.6). The sampled P-RAMS suggest a high rate of response to questions; 89.4 percent. However, “other” risk factors are rarely added by staff, if they have arisen. IEG analyzed data on underlying risk identified in the P-RAMs questionnaires. Figure 3.7 describes, for individual risk areas, the proportion of responses that indicate that implementing agencies do have adequate capacity to deal with the specified risk. Thus, a higher score indicates higher capability, and hence, lower risk. Among the higher risk factors identified are bid advertising; the bid evaluation and award area (which at 57 percent “yes” had the lowest ratings); staffing capacity; and record keeping, process clarity, and agency accountability. The pattern of information contains both similarities and some contrasts with the pre-P-RAMS analysis: implementation capacity (mainly staffing issues) is a common thread. In both, procurement oversight and contract management were among the less risky areas— which raises a query for proposed additional emphasis on this area. Because of limited sample sizes, these comparisons should be interpreted with caution. They do not suggest that P-RAMS has added any particular insights, although systematic analysis of the data—not only at the level of individual projects, but strategically aggregated by region, sector and over time—may have findings. 110 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Box 3.6. Perceptions on New Risk Management Tools—P-RAMS and ORAF While perceived as useful for raising overall awareness of procurement risk as part of project design and administration (Morocco) and modestly effective for identifying risk through early detection (Azerbaijan and Bangladesh), views were more guarded in terms of its effectiveness, with cautions regarding too mechanistic an approach. In Bangladesh, P-RAMS was perceived as being “too mechanical”—a view shared by some Bank task team leaders who see P-RAMS as an additional compliance tool that adds little value to project outcomes. Furthermore, P-RAMS was perceived as failing to pick up important issues reflected in the Project Concept Note stage of the process, pointing toward integration issues between procurement staff and task team leaders. In Ethiopia, Indonesia and Tanzania, P-RAMS was perceived as useful conceptually, especially by country clients, but not very helpful at tackling fraud and corruption risk, especially in contract implementation. In the Philippines, P-RAMS was criticized for being “too subjective and not evidence based.” In Senegal, P-RAMS was also criticized for not being user friendly. In turn, ORAF was perceived by country clients as only being relevant to the Bank and not to executing agencies. In Ethiopia, Bank task team leaders perceived ORAF as being “too theoretical” and failing to address “real risks.” Bank procurement staff in Tanzania view it as too generic, not adequately differentiated across projects. Country management seems to have shared this view. In Indonesia, ORAF was perceived as “something that was a good idea but is no longer effective.” It was deemed to be “…so inclusive and comprehensive that almost everything is seen as a risk….” Thus issues that are truly important are not highlighted. Finally, from the point of view of procurement, and the integration of procurement into the overall risk management framework, Bank procurement staff and Bank task team leaders in Indonesia, Morocco, and Peru perceived ORAF as lacking a comprehensive section on procurement, and procurement staff in Turkey also considered ORAF not particularly effective for procurement risk management. Source: IEG field visits. Risk-Mitigation Action Plans under P-RAMS A principal rationale for introducing the P-RAMS tool was to help ensure a more focused mitigation plan, including specific responsibilities and targeted timelines for mitigation actions, to address the main risks identified. A qualitative reading of the sampled 30 mitigation action plans does reveal that many mitigation action plans indeed contain well-specified mitigation measures and accompanying risk ratings for each risk factor, 111 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY completed for each agency and in some cases for several agencies. A health project in Brazil had 17 agencies (with three P-RAMS sequences) and associated mitigation action plans. Also, where the risk questionnaires show higher risk ratings, more items are specified in the mitigation action plans, as would be expected. Figure 3.7. Summary Results from P-RAMS Applications, 2010–12 Source: IEG analysis of P-RAMS questionnaires. Note: P-RAMS = Procurement Risk Assessment Management System. However, there was also a significant degree of variability or even mismatch in the placing of mitigation measures. In many cases where a risk factor was rated high or substantial in the risk questionnaire, there were no mitigation measures, and others in which mitigation was addressed to risk factors rated moderate or low, as evidenced in the data by the relatively loose correlation between the two instruments (Figure 3.8). These findings raise questions as to whether the goal of “sharpening the focus” of risk mitigation is being well met. Figure 3.8. P-RAMS Risk and P-RAMS Risk Mitigation Source: IEG analysis of P-RAMS data from 2010–12. Note: P-RAMS = Procurement Risk Assessment Management System. 112 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY IEG undertook additional analyses of successive P-RAMS to see whether the differential between initial and residual risk increased (that is, whether risk declined) over time (Appendix C). Results show that differentials are largely static over the periods of time for which observations are available. If, for example, initial and residual risk are reported as high and moderate in the first P-RAMS sequence, subsequent sequences show similar findings. ADDITIONAL OBSERVATIONS ON P-RAMS • P-RAMS operational links remain limited. Although PADs do reflect risk issues pointed out in the P-RAMS template, acknowledgement of P-RAMS is rare: of 22 PADs examined for the period 2010–12 only one gave mention to P-RAMS in the PAD. Fifteen mentioned and made use of ORAF. Systematic linkage of information (including Post-Procurement Reviews when available) will enhance the effectiveness of P-RAMS as a risk-management tool. • P-RAMS risk hierarchy is unclear. Project risk, in P-RAMs, refers to procurement risk focused on the implementing agency. However, project risk also embodies other (nonprocurement) risk elements and the separate treatment of procurement risk, as an element of all risks facing a project may be preferable and reduce overlaps with risk concepts in ORAF. • P-RAMS, as a tool to address fraud and corruption risk, could be sharpened. Although the P-RAMS Risk Assessment Questionnaire does have, among its 60 questions, some that explicitly refer to fraud and corruption–related issues (five in all): they are subsumed under a number of separate areas, which reduces transparency once they are aggregated. 119 However, focus could be more potent with a specific risk factor related to fraud and corruption. Compared to the enhanced treatment of fraud and corruption risk in project PADs, already evident in after 2007–08, it is not clear that P-RAMS has added much value in this area. • Addition of new issues, aggregation. The fixed template may add rigidity: although there is a “custom” facility to add additional risks, this does not seem to have been used by the procurement specialist. Moreover, the template weighs all risks equally. But risks need to be prioritized, and aggregation based on equal weights can give a misleading picture (as pointed out by field staff, in instances where a single high risk has significant weight). Though the template is modifiable in principle, changing weights is not transparent. Finally, there are three questions where a “no” response actually denotes a positive feature, which also leads to some difficulties in aggregation that could be easily addressed. 120 To summarize, P-RAMS offers a useful standardization of risk factors and risk-mitigation plans and provides an ongoing tool of dynamic content to monitor procurement risks throughout the project life-cycle. Althought it is still too early to draw definitive conclusions, there are indications that some improvement may be needed. Despite its short lifespan, some evidence points to a lack of adequate matching of higher risk and 113 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY mitigating actions. Sequencing is clearly important, but the spacing should be kept realistic, with due regard to efficiency. There are questions about whether the whole P-RAMS process is overly cumbersome for the benefits it brings. Both risk assessments and mitigation plans were highly developed before P-RAMS was introduced. Consideration should be given to making some minor but important modifications to the template, to allow for greater flexibility in risk coverage, permit a relative weighting of risk factors, reorient certain questions, and reduce aggregation issues. Consideration could be given to formulating a separate risk factor for fraud and corruption, to give more emphasis to those risks. Integration of P- RAMS applications with other operational tools (ISRs, Post-Procurement Reviews, ORAF) and more systematic use of the information it generates could help not only individual projects but also broad strategic directions to be taken by Bank procurement. Measuring Risk Failure—Misprocurement, Complaints, and Investigations Among the Bank’s armory of instruments to guard against procurement risk are the tracking of misprocurement and procurement complaints; as well as preventive and investigative work undertaken by the Bank’s Integrity (INT) Vice Presidency. IEG’s brief review of evidence in each of these areas suggests that the Bank is currently operating under broadly acceptable levels of procurement risk. MISPROCUREMENT AND PROCUREMENT COMPLAINTS Each year the Bank has a number of contracts under which misprocurement is declared, which arise from a variety of sources. Bank data on misprocurement, measured against the total number of contracts awarded annually, show that misprocurement rates represent 3–3.5 cases per thousand, which appears low. 121 The Bank also maintains a central complaints database, made by bidders (mostly losing bidders) in relation to contract awards. 122 Following recommendations made under the IDA 14 Internal Controls Review the Bank has improved the quality and rigor of its central complaints database, which is now kept current, including mandatory recording of all case closings. Data show that the number of complaints has varied little from an annual number of 300- 400 over the past decade, on all contracts, whether prior reviewed or not. When standardized by the number of procurement contracts awarded each year, the number remains quite stable at about 2 percent of the prior-review contracts, or below a half percentage of all contracts per year.123 One issue with complaints is their potential to impose delays in the procurement process. The data for complaints in the database in FY12 show that the average time for resolution of complaints has been about 150 days, though some were resolved sooner than that, and 114 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY a significant number have taken much longer to resolve. Figure 3.9 shows the distribution of cases according to the length of time to their resolution. IEG’s discussions in field visits indicated that in some cases frivolous complaints may be lodged by competing bidders, as a deterrent to competition. More speedy dispatch of such complaints would improve the process. With rates of complaints—which are a normal part of the process—at such low levels, there is little evidence to suggest that there is much risk failure involved, though the speed of resolution is an area for possible concern. Figure 3.9. Time Distribution of Complaints Resolution Source: World Bank database. Note: 2,157 observations. FRAUD, CORRUPTION, AND THE ROLE OF INT To what extent is there evidence of fraud and corruption risk in procurement, and what risk controls address this? One indicator is the number of “red flag” referrals that the operating regions have made to INT in recent years. Once INT receives a “referral,” it is treated as a complaint. Complaints number around 500 per year. Though seemingly high, not all of these complaints are evidence of control failure, or of fraud and corruption. 124 Furthermore, the complaints relate to both prior-review and postreview contracts. Given that the total number of prior-review contracts per year is around 18,000 (2011), alleged fraud and corruption represents a very small percentage of prior review contracts. In addition, a complaint received in one fiscal year may relate to a prior review contract awarded some years before. Yet some of INT’s Detailed Investigation Reports show that even when the Bank’s procurement procedures are adhered to, and even in ICB contracts prior reviewed by the Bank, fraud and corruption have sometimes been found. 125 Detailed Investigation Reports have detected instances of possible fraud and corruption that are significantly greater than what has been detected by the government or the World Bank, illustrating the limitations of enhancing fraud and corruption controls through tighter procurement alone. 115 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY INT’s Preventive Services Unit offers tools and training to the operating regions in managing fraud and corruption risks in Bank projects, based especially on its Fraud and Corruption Awareness Handbook, the Red Flags Toolkit, which alerts task team leaders and procurement specialists to a number of potential fraud and corruption risks in the procurement process, although some corruption flags may also arise at other stages or during contract management. 126 Some evidence collected by INT as part of a review of Final Investigative Reports and Active Investigations shows that most (89 percent) of identified irregularities (red flags) occur as part of procurement, defined as initial procurement notice to contract award. 127 One percent occurred during project design, and 5 percent each during contract management and financial management, albeit based on less information on contract management, as this is beyond the purview of the Bank. IEG partnered with INT to undertake an investigation of the extent to which complaints matured into cases deemed worthy of further investigation—around 150 of the 500 or so annual complaints. According to INT, the decline in the number of cases per year (Table 3.7) is due to ongoing efforts by INT to resolve more cases at a preliminary stage prior to full investigation and efforts to redirect limited investigative resources toward cases with a greater likelihood of significant impact. Table 3.7. Data on INT Investigation Cases by Region Conducted in FY09–12 Operating Regions (and IFC) FY09 FY10 FY11 FY12 AFR 36 55 21 30 EAP 23 28 7 8 ECA 28 30 12 17 LAC 10 24 9 8 MNA 7 18 5 6 Other (INT/IFC) 1 5 4 TOTAL 139 195 73 80 Source: IEG analysis of INT data. Note: Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa. INT = Integrity Vice Presidency. The recent INT review of its final investigative reports shows that the incidence of fraud and corruption varies according to the type of misconduct. Fraud (that is, cases where bidders claim to meet the criteria set out in the bidding documents, when they in fact do not) appears to be the dominant mode. Corruption (that is, bribing or otherwise inappropriately influencing officials) appears also to be evident in about half the reports reviewed. Collusion (that is, anticompetitive arrangements between bidding entities) appears also to be present but much less frequent. 128 According to INT, such misconduct often goes hand in hand with corruption. IEG further undertook a review of individual complaints in the INT database for FY10 and FY11 to classify complaints at the intake phase in INT according to the stage of the 116 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY project cycle to which the complaint referred (Figure 3.10). 129 Among identifiable procurement stages, the bid submission stage of the cycle had the highest number of complaints; these mostly involved allegations of collusion among the bidders. Fifteen percent of the 2010 complaints, 18 percent of the 2011 complaints, and 17 percent of overall complaints referred to the bid submission stage. Complaints at the contract award stage of the procurement cycle were the second highest. These complaints usually related to accusations that the contract was awarded to a bidder who was not necessarily the lowest bidder or one who did not meet the specified technical criteria. Echoing findings from P-RAMs, it was found that just 10 percent of the complaints across the two years were related to contract award, and 5 percent per year, on average, during contract implementation. A caveat regarding these findings is that 17 percent of complaints across the two years that were deemed to be procurement related could not, by the nature of the complaint, be attributed to a specific stage in the procurement cycle. Figure 3.10. INT Complaints by Stages in Procurement and Project Implementation Source: IEG analysis of INT data. Note: INT = Integrity Vice Presidency. INT investigations have found that some identifiable red flags may go undetected, sometimes through the inexperience of project implementing unit or Bank staff, or concerns about raising sensitive issues, such as a perceived need for more evidence or possible implications for staff if they report the matter to INT. INT’s draft paper on potential red flag issues offers a number of suggestions to operations staff on how to improve their use (Appendix Box C.6). Of relevance to the present evaluation is their likely recommendation to better define a risk basis for transactions, as opposed to a value based threshold, as suggested by IEG (see section on threshold analysis earlier in this chapter), and to build capacity in implementing agencies to detect the most common indicators of fraud and collusion. 117 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY IEG undertook a further exercise on INT data on cases under investigation, over the years FY10 and FY11, to investigate the extent to which it may be possible to predict those cases likely to be identified by INT as fraud and corruption cases, by reviewing the extent to which PAD risk ratings (overall project risk, procurement risk, and financial management risk, as identified in PADs) are correlated with INT cases—especially PAD procurement risk ratings. Data were available for 350 cases that provided at least one risk rating. Table 3.8 provides the frequency and percentage of available PAD risk ratings in each category. Table 3.8. Correlation of INT Cases with Project, Procurement, and Financial Management Risk Risk rating (number) Frequency Percent Cases with a project risk rating (296) High 33 11 Medium 261 88 Low 2 1 Cases with a procurement risk rating (318) High 176 55 Medium 133 42 Low 9 3 Cases with a financial management risk rating (243) High 54 22 Medium 177 73 Low 12 5 Source: IEG analysis of INT data. Note: Based on 350 cases that have at least one risk rating (project, procurement, or financial management). As shown in Table 3.8, the majority of cases (318 of 350) had a procurement risk rating, and more than half (55 percent) were rated as having high risk. By contrast, the majority of cases that had project or financial management risk ratings reported them to be medium— 88 percent and 73 percent, respectively (Figure 3.11). The bulk of procurement risk ratings fell in the high or medium categories. These findings suggest that levels of procurement risk ratings are somewhat more closely associated with potential fraud and corruption risk than with project or financial management risk. Together with the findings of Chapter 1 that good procurement outcomes are correlated with high levels of development effectiveness, risky procurement may also imply a greater risk of irregularities. 118 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Figure 3.11. Correlation of INT Cases with Project, Procurement, and Financial Management Risk (cases with at least one risk rating) Source: IEG analysis of INT data. Note: PAD risk ratings of INT cases—350 cases with at least one risk rating. SANCTIONS, SUSPENSIONS, AND CORPORATE RISK Bank lending processes require task teams (procurement specialists or procurement accredited staff and task team leaders) to check information on suppliers, as part of the due diligence process, prior to making procurement decisions. 130 Mandatory among these are the list of suspended firms, available to Bank staff on the intranet and to Borrowers via Client Connection, and the list of debarred firms, which is public. Firms on these lists are ineligible to receive Bank-financed contracts; firms debarred for more than one year are also subject to sanction by other MDBs, under a cross-debarment arrangement (some debarments therefore originate from other MDBs). 131 The lists of suspended and debarred firms are maintained by the Bank’s procurement anchor and are updated by the Office of Suspension and Debarment, the Sanctions Board and INT to reflect the results of the Bank's administrative sanctions proceedings. The Bank has a two-tier sanctions system for the review and disposition of cases arising from INT’s investigations. The Office of Suspenstion and Debarment represents the first tier of the system, and the Sanctions Board represents the second (appellate) tier which has the function of a higher authority in cases where the decisions by the Office of Suspension and Debarment become contested by respondent companies. 132 Since 2009, INT has been permitted to request that the Office impose an “early” temporary suspension on the subject of an ongoing investigation for up to six months, with the possibility of extension to 12 months, if it determines that there is already sufficient evidence that the firm engaged in sanctionable misconduct. 133 Box 3.7 provides a summary of key features of the debarment process in operation, including with selected data for adjudications by the Office of Suspension and Debarmant in recent years. The sanctions process is an outcome of the 150 or so investigations that INT decides to pursue from the 350–500 complaints it receives each year. As shown in Box 3.7, in recent years the cases have resulted in an average of 47 suspensions and 44 debarments each year. Cases may be completed within a year, but in 119 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY recent years less than 1 percent have been completed within one year, while 25 percent took between one and two years, and 74 percent took more than two years to complete. Box 3.7. The Office of Sanctions and Debarment—Two-Tier Sanctions System The two-tier Sanctions Board (about 40 percent of cases) mechanism External members (4) + Bank staff (3) Independent review (not bound by OSD) Decisions are final, no appeal OSD (about 60 percent of cases) Evaluates evidence submitted by INT in SAE Issues notice of sanctions proceedings to respondent Temporarily suspends respondent Sanction becomes effective if respondent does not contest Process There is a strict division of function between INT (investigation only) and OSD/Sanctions Board (adjudication only). INT decides, after investigation reveals sufficient evidence, to send an SAE. If OSD finds that evidence is lacking (overall, or for specific allegations) it may send the SAE back to INT for further investigation. Once evidence is agreed by OSD, OSD sends notice of proceedings to the respondent. If respondent wishes to contest (within 30 days) it must send a letter to Sanctions Board. Five forms of The standard form of sanction is debarment from eligibility to be sanction awarded World Bank contracts, the default penalty being a three-year debarment, with conditional release. Mitigating factors (for example, severity of infringement; cooperation with INT investigation; internal reform and/or severance of offending personnel) can reduce the term of debarment, based on prestated criteria and amount of relief. The five forms of sanction are • Debarment • Debarment with conditional release • Conditional nondebarment • Public letter of reprimand and restitution • Negotiated resolution. Where there is compelling evidence of sanctionable activity the OSD may impose an Early Temporary Suspension on a company/individual. In such cases the respondent’s name does not appear on the debarment list. This creates a “grey area” which may lead to confusion over whether a suspended party may or may not be awarded a contract. 120 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Statistical outcomes FY 09 FY 10 FY11 FY12 FY13 Average (FY09–13) Case load Cases sent by INT to OSD 39 43 27 25 33 33 Cases OSD sent to respondents 10 29 33 33 25 26 Settlement agreements - - 11 16 8 Sanctions results Temporary suspensions 32 51 55 58 41 47 Debarments/other sanctions 13 45 34 83 47 44 Duration of cases Mean duration (days) 945 716 433 535 465 < 1 Year Years >2 Years TOTAL Cases submitted to OSD (09-13) 9 41 112 162 Source: OSD statistics. Note: INT = Integrity Vice Presidency; OSD = Office of Suspension and Debarment; SAE = Statement of Actions and Evidence. INT, on behalf of the Bank, also maintains a Company Risk Profile Database—a list of suppliers against which complaints have been received and a case opened, although not concluded. As part of the commitment to due diligence, prior to issuance of no-objections for prior review contracts, staff are mandated to check potential suppliers against the Web-based database. This process seems to suffer from errors of commission and omission. On the one hand, small differences in the way a name is entered can lead to false negative results. On the other hand, if a positive result ensues, the action required of task team leaders is not clear. Technically, such firms are not debarred by the Bank. Thus, task team leaders are not prohibited from approving contracts to suppliers on the database but are advised to consult INT to get more insights. If such additional review points to a high level of concern, a decision may be taken to withhold a no-objection. Staff point out that the process of obtaining feedback from INT can add weeks to processing time. Overall, limited integration of the separate checks that need to be made—the debarment list, temporary suspension list, company risk database, and anti money laundering—add to cumbersome and slow processing. 134 INT affirms that the Corporate Risk Profile Database will be enhanced in FY14, to include a more robust search engine along with being integrated into the forthcoming electronic no-objection system. Links between INT and Bank operations were recently audited by IAD. IAD was of the opinion that despite initiatives taken to improve outreach and feedback, weaknesses remain in terms of the use of information from INT investigations to feed back into operational learning, and the Bank is still not using its INT-generated knowledge to enhance fraud and corruption risk management to best effect (World Bank 2013b). 121 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY PERCEPTIONS OF THE ROLE OF INT—IEG FIELD VISITS Field visits described the frequent occurrence of differences between the Bank’s list of debarred firms and the list of firms debarred under national sanctions regimes. The eligibility of such firms to bid on Bank-financed projects has been a frequent concern to client governments (for example, in Ethiopia, Mexico, Tanzania, and Turkey). Yet private sector suppliers often prefer the use of the Bank’s list of debarred firms, especially when country lists are long. In Turkey, debarred firms number some 3,000 and reasons are broad based and include performance factors. This raises the wider question of whether the Bank should use performance factors to screen potential contractors and suppliers. Task team leaders point out that even if not debarred, or under a “negative list,” it may be worthwhile for the Bank to maintain a “positive list” of good performers.135 The issue of firms that are under investigation by INT, or within the World Bank Office of Suspension and Debarment process, but not yet debarred, also arose. Staff prefer to wait until uncertainty ends, often leading to protracted delay. However, this raises the possibility of unreasonably excluding firms that may not be wrongdoers and also leads to the awkwardness of explaining the situation to clients as the firms are not officially debarred, leading to complaints of arbitrariness on the part of the Bank (as occurred in Turkey and Bangladesh). 136 As is noted on the CRPD website, “The existence of the name in the database is not a basis to withhold the issuing of a [no-objection letter].” By contrast, ADB does not maintain such a list and so does not have this problem. ADB does have a nonperforming contractor list. However, the intrinsic effectiveness of debarred lists has also been challenged. In Indonesia, respondents observed that the debarment process has limitations, as firms can regroup under a new name. Some respondents were critical of INT and its contribution to identifying and mitigating procurement risk, especially its sense of proportionality and judgment in terms of “what is important,” particularly if a project is delayed by the actions of INT. In contrast, the preventive aspect of INT’s work has been more appreciated than its investigative function. Staff point out that the process of obtaining feedback from INT can add weeks to processing time. Overall, limited integration of the separate checks that need to be made—the debarment list, temporary suspension list, the company risk profile database and anti-money laundering—results in cumbersome and slow processing. Finally, many respondents, especially those from the private sector, raise questions as to how much can be achieved from project-level or transaction-level controls. Even if issues are detected and corrected in the context of specific transactions, they could well occur in 122 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY others. In some environments, where issues of governance and fraud and corruption are endemic, the Bank could consider more system-level engagement, for example, through country-level dialogue or programs that need not be linked to procurement. 123 4. Achieving Efficiency in Bank Procurement An important aspect of overall efficacy of the procurement process is the extent to which it makes efficient use of resources. Although it is recognized that the review of procurement by borrowers goes beyond compliance towards a policy and quality review, timeliness, process efficiencies and delays in procurement have been raised as a prime concern in country consultations, undertaken by both management and IEG. IEG’s review and analysis of some factors that affect the process efficiency of Bank procurement identifies factors that will remain relevant to the procurement function and framework. Procurement Efficiency and Value for Money—Tracking Procurement MAIN FINDINGS  Currently, Bank procurement tracking systems are not equipped to provide key information needed to monitor procurement objectives of economy, efficiency, risk management, transparency, or value for money. The three regional “tracking” systems each have different objectives and architecture, as well as their own merits and limitations. For those regions that do not have a tracking system, practices are diverse and data are mostly maintained in field offices. Collecting unified Bank-wide information on procurement parameters is difficult.  Nevertheless, at a Bank-wide level efforts have been made to develop individual modules of a future system architecture. Today there is a renewed agenda within the procurement anchor at the Bank’s Operations Country Policy and Services Vice Presidency to integrate aspects of these systems.  IEG’s analysis of procurement process efficiency suggests that average time taken is clearly much longer than Bank standards, but also, that there is a high level of variability in processing times, with a “long tail” of contracts that take considerably longer than average times. There is variation across procurement methods. National competitive bidding, even when prior reviewed, is notably quicker than ICB, and conversely, consultant contract processing through quality- and cost-based methods is particularly time consuming. The size of a contract, in terms of its value, is a significant determinant, and country capacity and governance appear to matter.  Looking ahead, new procurement methods such as framework agreements can offer a means toward increasing efficiency, not only for the Bank but also for its clients. Many Bank client countries have introduced provisions for framework agreements in their own procurement systems. Although the 124 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Bank introduced the use of framework contracts from 2011, in practice these have been little used.  E-procurement can offer considerable scope for improving procurement efficiencies, among other things also increasing transparency and lowering potential for corruption. Such platforms have won global acceptance and are being incorporated into new procurement legislation in jurisdictions such as the European Union. TRACKING CONTRACT-LEVEL PROCUREMENT DATA As pointed out in IEG’s field surveys, and in management’s proposals for new directions in procurement, one of the core issues raised by clients and staff alike concerns Bank procurement efficiency—notably, the time taken in the Bank’s procurement process and the frequency of delays. This motivates IEG’s review of procurement tracking systems and their data. Second, used effectively, contract data could collect information not only on the contract process but also on the prices paid and whether the Bank is achieving value for money. Bank lending-generated contract data on transactions and prices could provide a wealth of data if properly harnessed. IEG therefore reviews Bank procurement tracking information systems and the data they yield to evaluate the extent to which procurement monitoring objectives are met. Specific monitoring objectives would include the following: • Fulfilling Bank fiduciary objectives—ensuring that funds are used for intended purposes and contributing information needed for disbursement 137 • Making informed choices about markets and suppliers, for example, for setting (methods) thresholds • Monitoring the extent to which core principles of procurement guiding the process have been used—considerations of competition, economy and efficiency; transparency and equity; and domestic market development • Providing management information on the performance of the procurement process and identifying bottlenecks in procurement execution (elapsed times, clearance levels and sequences); project execution (proportion of expenditures contracted/disbursed; “burn” rates); and agent execution (tracking the client/borrower, task team leader, procurement specialists, and private contractors, if the system embraces contract management). • Enabling borrower/client monitoring of procurement and project execution • Increasing market transparency and price discovery—generating information for a wider group of market agents with the potential of getting better value for money, not only for Bank projects but also for overall public sector efficiency in client countries and for other agencies of development (as in the example in Box 4.1). 125 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Box 4.1. Benchmarking Municipal Procurement Activities—An Example from Finland In their 2003 study Kivistö, Virolainen, and Tella gathered procurement price data from nine municipalities and three hospital districts in mid-sized Finnish cities and some of the countries’ larger rural communities, to analyze differences in procurement outcomes between these public organizations. Though most of the organizations claimed to have a procurement strategy, the report shows that when interviewed, they clarified that this meant merely the existence of procurement guidelines. Competitive tendering was used in two hospital districts and two municipalities, with only two organizations publishing requests for proposals on their Internet home page. Procurement Price for Peeled Potatoes in Different The study analyzes, for Municipalities and Hospital Districts example, prices of paper Purchasing Price Quintiles towels and finds big price 100 differences even for Purchasing Price Quitnile 80 identically specified 60 products—that is, after 40 adjusting for either one-ply or 20 two-ply paper. 0 Another example looks at the 1 2 3 4 5 6 7 8 9 10 purchasing price of peeled Source: Kivistö & Virolainen 2003 Municipal Public Entity potatoes. The results show variations in purchase prices ranging from 10–180 percent—with procurement volume not being a particularly significant determinant. The study indicates that variations in procurement competence are a main factor behind these results, depending on whether tendering is purely operational/opportunistic or also strategic. Benchmarking showed that major differences can exist across organizations, on a scale beyond what can be explained even by logistics costs. Benchmarking of different types of data can take many forms at an organization such as the World Bank: internal best practices; best practice in other MDBs/IFIs; best practices among a supplier pool in a country /region; benchmarking outcomes in different organizations within a client country; and benchmarking using different procurement methodologies and criteria, and so forth. Sources: Kivistö, Virolainen, and Tella 2003; Thai and others 2005; Korpela and Tuominen 1994. 126 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT IEG finds that the Bank’s current procurement information systems are far from being able to fulfill the above objectives. Although some basic fiduciary information is collected (on prior review contracts only, through Form 384), the Bank does not have a unified central system to track, organize, and report on its procurement process, to monitor its efficiency, or to centrally track rates of implementation. In recognition of the need for such systems, three of the Bank’s six regions (Latin America and the Caribbean, Africa, and the Middle East and North Africa) launched their own regional systems around 2008, each of which partially contributes towards the range of monitoring needs. None has the ability to provide all the above data, and all are very distant from the ultimate goal, albeit not necessarily an easy one, of capturing information on unit price or value, in a manner that would enable comparisons across contracts, markets, suppliers, or over time. Management recognized the Bank’s need for a centralized system to track contract data some years ago; initially efforts were made to build such a system, but these were abandoned in the hope that integration would come later as part of the modernization agenda. The Bank’s three regional systems each has different objectives and architecture, with their own merits and limitations. These regional systems are inherently unsatisfactory from the perspective of having a Bank-wide system to track procurement transactions. Other regions decided they would wait for a universal Bank-wide system to be developed. For those regions that do not have a tracking system, practices are diverse. Data are mostly maintained in field offices, sometimes by procurement staff and sometimes by sector units. Collecting unified information on procurement parameters is frequently difficult. Today there is a renewed agenda within OPSOR to move gradually to adapt and/or integrate aspects of these systems into a unified approach. This is clearly a priority. Further expansion is needed to include systematic information on price and value, if economy, efficiency and value for money are among the Bank’s procurement objectives (see Box 4.2). Current systems and their limitations are briefly described below, and a more detailed account be found in Appendix D. 127 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Box 4.2. Procurement Tracking Systems at the World Bank—Early Efforts In 2004–06 a pilot document submission system was created inside the Client Connection portal. The pilot was abandoned for a variety of reasons: system speed (client systems were too slow); system reliability (data quality); difficulties in client interface with Bank document filing systems; the likely cost of developing such systems, among competing resources, and prioritization given to developing other key system flags and variables. Coming at a time (following the 2005 Paris Declaration) in which it seemed clear that the Bank was moving towards adopting the use of country procurement systems, moving away from the notion of a unified Bank-wide procurement information system appeared to logically complement an overall move away from centralized procurement and controls. At the same time, the Bank opted to end “contract accounting” in the Bank's Loan Department, whereby disbursement was linked to specific awarded procurement contracts. This further diminished OPSOR incentives to maintain central procurement information systems. OPSOR made a decision to confine its focus to simplifying the collection of procurement data in its Form 384 and to focus its efforts on developing individual procurement system modules (for example, the postreview system; a complaints database; select INT databases; and an integrated Procurement Advertisement and Award Notice system). Source: IEG interviews. BANK-WIDE TRACKING OF PROCUREMENT: FORM 384 Since 1988, the principal Bank-wide source of procurement information tracking has been the Bank's Form 384, a Web-based interface for entering contract information for IDA/IBRD Bank-funded contracts (Box 4.3). 138 A key purpose of the Form 384 has been fiduciary; to ensure that a no-objection and signed contract exist before disbursements are made. Form 384 also records the procurement method and contract award information including supplier names, nationality and eligibility status. The form has undergone several revisions since data collection started, which have successively reduced the amount of information collected. As a tool for meeting the objectives described above, it has significant limitations. To begin with, it does not cover all contracts but only those at or above the applicable prior review threshold. In addition, the link between procurement and contract based disbursement was further weakened from 2007 when the Bank's loan department abandoned the concept of contract accounting (Box 4.3). Although the Bank can declare untoward disbursements as ineligible, it does draw the fiduciary potency of Form 384 into question. 128 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Although contract award data on the winning contract are available to the public after contract signature, some fields of data remain restricted and are available only to Bank staff, not to the borrower or to other external parties. This tends to limit its potential contributions to transparency or to facilitate Bank/borrower joint participation in implementation of project procurement plans. Further, in terms of contract or project processing, it captures only the final stage of the procurement process: the date of the Bank's no-objection to the contract award and the date of contract signature. For this reason it does not provide information capable of tracking processing time at different stages of the procurement cycle, a requirement vital to measuring efficiency. It does not capture unit price data. And in terms of fiduciary objectives, the form does not separate the part of a contract that is Bank financed, compared to other sources, but reports instead on the total value. To the extent that the borrowing government or other donors finance a part of the contract, the role of the Bank is not captured. Finally, questions have been raised about the quality and reliability of its information. Box 4.3. Procurement, Disbursement, Contract Accounting, and Form 384 Prior to 2007, the Bank used contract accounting, that is, it monitored payments made against contracts awarded. In 2007 the decision was made to abandon contract accounting, because of the time consumed in manual checking of payments contract by contract, and with due consideration to the respective roles of the Bank’s Controllers Office (which is not an accounting office) and the borrower. Disbursements are tracked not by individual contracts but by categories of expenditure. If a category has headroom for further disbursements, payment for a given contract can in principle be made. In one recent case, significant overpayment occurred on a contract in Kosovo. The question was raised whether the 2007 decision might have led to a systemic control issue in procurement risk management. The Bank Controller’s Office points out that though serious, this occurred because of a series of misunderstanding at several steps in the procurement-disbursement chain that confused two contracts with a consultant in the same country. A number of controls that were in place failed, on the side of the client as well as the Bank. Further, as frequently happened, Form 384 was not updated and kept current as to payments made. Concerned that the episode might reflect a systemic issue, the Controller’s Office conducted a survey of 236 contracts to check their disbursement accounts for irregularities. They found only one that was not in full compliance—that is, funds had been overdisbursed—evidence they took to be satisfactory of systemic security. The Controller’s Office does not deny that overdisbursements are possible under the current system. 129 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT IEG believes that the 236:1 survey result is encouraging and suggests that this is not a widespread case of risk failure, though monitoring is desirable. As to remedies, cases of this sort would be detected as part of the financial management and audit process, adding impetus to the idea that the Bank should encourage tighter integration of procurement and financial management processes. Source: IEG interviews. EXISTING REGIONAL SYSTEMS The three regional systems that have been developed since 2007 comprise: the Procurement Plan Execution System (SEPA) in the Latin America and the Caribbean Region (with implementation expanding to some countries in Europe and Central Asia), the Procurement Portfolio Dashboard in the Middle East and North Africa Region, and the Procurement Cycle Tracking System in Africa (Table 4.1). SEPA: Procurement Plan Execution System SEPA, launched in 2007, is a Web-based system that focuses primarily on the monitoring and execution of procurement plans related to Bank-funded projects. 139 SEPA’s objective is to promote transparency in Bank operations and to offer a procurement management tool to the Bank and to borrower governments. A part of the SEPA system is open to the public—its information on procurement plans and contracts financed for Bank projects. 140 Fourteen countries in Latin America and the Caribbean and four in Europe and Central Asia have implemented SEPA for use with World Bank-funded projects; work on pilots for India and Brazil is under way. 141 In some countries, for example, in Argentina and Nicaragua, use of SEPA is included in the loan agreements for selected projects. SEPA is shared with the Inter-American Development Bank, which has helped its uptake in Latin America. As a management tool, it helps both the Bank and its borrowers to ensure that detailed procurement plans are prepared, and to identify variations between planned and actual procurement activities. SEPA is a stand-alone system and integrating its information with other tracking systems (Form 384, SAP) can be burdensome, as staff have to enter data multiple times. Data in SEPA are populated by the borrower; although some data are mandatory for procurement plan approval, other data hinge on the readiness of the borrower to enter additional information, and there is a high level of variability in information by project and country. Yet its external availability and especially the sharing of the system between Bank and borrower give SEPA a positive rating for transparency and for building client ownership in the procurement process. 142 130 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Table 4.1. Summary of System Contributions to Tracking Procurement Performance PR performance objectives F384 SEPA PROCYS MNA DASH Fiduciary Discipline YES NO NO NO (Disbursements for But partial (only prior Not a disbursement tool Not a disbursement Not a disbursement tool intended purposes) reviewed contracts); tool weakened since 2007 Risk indicators NO NO NO YES PPR and other risk ratings Transparency NO YES YES NO No public access; not all Public section But partial: prior Portfolio based not transaction contracts Borrower access review contracts only based; access only to Bank staff Efficiency (Processing NO YES, but YES NO time) Tracks only final stages at Can track processing Designed specifically Not transaction based contract signing time but only if Borrower to track processing enters actual dates times Economy, competition YES limited. YES NO NO Information on bid award information on bidders to by country of origin the extent borrower provides Client/borrower NO YES YES NO monitoring Not available to the client Designed to be Designed as Designed as in-house interactive with Borrower interactive tool management tracking tool including with borrower Price discovery NO NO NO NO Contract management NO YES YES YES Updating of procurement Tracking of ISR trends PPR compliance rates and plans results Domestic market NO NO NO NO development Overall procurement NO YES NO YES performance Status of procurement Has aggregate portfolio plans tracking Source: IEG. Note: MNA DASH = Africa Procurement Portfolio Dashboard; ISR = Implementation Status Report; PPR = Post-Procurement Review; PROCYS = Africa Procurement Cycle Tracking System; SEPA = Procurement Plan Execution System. Procurement Cycle Tracking System—Africa Region The Procurement Cycle Tracking system developed in the Africa Region is a platform of communications between the principal parties involved in the procurement process—the borrower, the Bank task team leader, and Bank procurement staff—on procurement processes for contracts subject to prior review. Each interaction or stage in the process is recorded, in terms of the number of days taken from the previous to the present stage. It thus tracks, for example, not only the total elapsed time between a borrower's first request for a no-objection and receipt of the Bank's final no objection, but also the numbers of iterations between the task team leader and borrower, between the team leader and different levels of procurement staff; from the 131 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT field procurement specialist to the regional procurement manager and the Central Procurement Board. The system was piloted in late 2008 and has been in use since 2009. It currently covers more than 460 projects in more than 40 countries in the Africa Region. The system is principally used as a management information system that measures responsiveness of different participants in the procurement process. It is partially integrated into Bank-wide systems, in that no objection notices and related materials are automatically filed into the Bank’s document database. However, its unique transaction request numbers cannot be linked to contract identifiers used in other Bank systems tracking contract data, such as Form 384. It is a Lotus Notes-based system and will face difficulties when the Bank moves away from this platform. Procurement Portfolio Dashboard—Middle East and North Africa Region The Middle East and North Africa procurement portfolio dashboard was also designed as a tool for the management of the Middle East and North Africa procurement unit’s resources. However, its emphasis, unlike the contract and transaction focused approach of the Africa system, is on the execution of the region’s loan portfolio and individual projects within that portfolio. Although SEPA and the Africa Region tracking system provide a platform for interactions between Bank staff and borrowers and the processing of procurement stages through the system, the Middle East and North Africa dashboard is not an interactive or automated system. Information is reported by Bank project team members, and staff in the Middle East and North Africa headquarters update the dashboard on a monthly basis. Unlike the other systems, it provides information on the overall status of project disbursements, on Procurement Post-Review complaints, ISR procurement ratings, and procurement risk ratings. It also provides information on the status of procurement plans (whether they have been updated or not), misprocurements, and cases sent to INT. Bank project team members’ time spent on supporting procurement related activities can be aggregated. This dashboard is available to Bank staff for download from its intranet site. FUTURE DIRECTIONS Given the fragmented nature of its procurement tracking systems, the uneven way in which they contribute to the Bank's procurement objectives, and other policy imperatives (fiduciary controls, the modernization initiative, and so forth), management has now committed to return to building an integrated tracking and controls system (Box 4.4). 132 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Box 4.4. Proposed New Systems Architecture: The Integrated Procurement Plan OPSOR has created a conceptual framework (a work program architecture) to help it build an integrated platform beginning with the integration of several interacting information technology systems. As described, the architecture rests on two pillars: alignment with the Bank's Modernization Initiatives and delivery of client-centric solutions. The key factors described on the side of modernization are: integration—of fiduciary and risk management systems in projects; risk management—with better tools for better and more timely quality control; transparency (open data, open contracting, big data); and knowledge management (delivery systems for continuous knowledge management and learning). The client-centric solutions include integrating data exchanges at the country systems level, including with other MDBs; flexibility (connectivity options between client and Bank systems); capacity building (developing an information technology toolkit for capacity building, that is, interactive bid documents); and contract management (implementing solutions to help clients execute contracts under Bank projects). The integrated procurement plan is intended to draw on existing individual tracking systems, plus new elements, such as an electronic no objection that will activate within Bank systems. At present this element is for the final no-objection only, prior to contract award, as tracked in Form 384. Whether it will later embrace earlier steps in procurement processing (that is, moving through decision stages) and data tracking remains to be seen. Tools will be applied on line in the Operations Portal, so once the data are entered and saved they are instantly also sent to related tools (ORAF, ISR, and so forth) tracked to integration points and archived. Some of these modules already exist, though their interlinkages are at best only partially operative. The integration project is in large measure an effort to bring those linkages into effective operation. Source: IEG interviews. Steps toward an integrated system began in November 2012 with the launch of a pilot of an electronic no-objection system, in two or more countries per region. 143 Piloting is scheduled to go to the end of FY13. The next step will involve developing an integrated document submission module. IEG supports the urgent need to move the new architecture forward. Yet integrating existing modules will not suffice for objectives that would help achieve value-for-money. Links with other Bank areas need also to be established (for example, linking risk indicators with INT data, and so forth) and it will be necessary to build some entirely new modules, especially in areas such as benchmarking of information and other elements of open data, open 133 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT contracting, and knowledge management. Efforts required to bring these elements together should not be underestimated. Periodic reviews of progress actually achieved would be desirable. Given the emphasis on sound monitoring in the Bank’s advice to its country clients, the Bank needs to position itself for a leadership role in this area. Measuring Procurement Efficiency—IEG Data on Sample Contracts As illustrated in the IEG case studies as well as in Bank management’s reviews, stakeholders in the Bank’s procurement process have been less critical of the Bank’s procurement rules than of its procurement processes. Delays in the execution of procurement are a core element of concern. IEG has therefore undertaken a first attempt to examine the extent of such delays and to understand the factors that increase the time taken to process a contract. IEG emphasizes that this is only one dimension, albeit an important one, of efficiency in the procurement process. IEG’s work is illustrative rather than conclusive, and apart from the findings, it demonstrates forms of analysis that may be useful for achieving economy and efficiency in procurement going forward. Given the fragmented and decentralized information on procurement processes across different Bank regions and field offices, IEG initiated its own request for data on prior-review contracts, based on a standardized template, in each field visit country. Data requested included, primarily, information on the dates of each step in the procurement process for which the Bank is required to provide a “no-objection” to a client, in order to be able to track elapsed time between and across successive phases of procurement (Table 4.2; field visit questionnaire). 134 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Table 4.2. “No-Objection” Steps for Prior-Reviewed Contracts Simple goods and works Two-stage goods and works Consultant services (ICB/NCB) (ICB/NCB) (QCBS) Specific Procurement Notice Specific Procurement Notice Expression of Interest Terms of Reference/Short List Draft Bid Documents Draft Bid Document Short List and Draft Request for Proposals Bid Documents as Issued Bid Documents as Issued Request for Proposals as Issued Bid Opening/Minutes Technical Bids Opening of Technical Proposals/Minutes Stage 1 Evaluation – Evaluation of Technical Proposals Report/Minutes/Amended Bid Documents Invitation to Stage 2 as Issued Stage 2 Bidding Opening of Financial Proposals/Minutes Bid Evaluation Report and Bid Evaluation Report and Combined Evaluation Report and Draft Negotiated Recommendation for Award Recommendation for Award Contract Signed Contract Signed Contract Signed Contract Contract Amendment Contract Amendment Contract Amendment (in some cases) (in some cases) (in some cases) Contract Completion Contract Completion Contract Completion Source: Bank procurement anchor. Note: Steps requiring a “no objection” are italic. ICB =international competitive bidding; NCB = national competitive bidding; QCBS = quality- and cost-based selection. The sample of 502 contracts obtained covers the period FY07–12; information received is concentrated around FY10–12. 144 Contracts for infrastructure account for a large proportion of observations by sector. By procurement category, data are distributed more evenly, with consultant services and goods and civil works each accounting for roughly one-third of observations. In terms of procurement methods, ICB contracts form around 40 percent of the sample, whereas NCB and quality- and cost-base selection each contribute about 22 percent of observations (Figure 4.1). 135 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Figure 4.1. Number of Observations by Country, Fiscal Year, Procurement Category, and Sector 300 By Sector 250 200 150 100 50 0 Source: IEG analysis of sample of contract data. ANALYSIS AND FINDINGS: AVERAGE ELAPSED TIMES To explore the time taken to process a contract, estimates of elapsed time were constructed between key procurement steps, from the Borrower Issue of Specific Procurement Notice to contract signature. Measured from the time when the borrower first submits draft bidding document to the Bank, results show that, looking at all contracts together, it takes on average 286 days until a prior reviewed contract is signed (Table 4.3). 145 Separating contracts by category, civil works take the longest (307 days), though there are similar overall processing times for goods (287 days) and consultant services (290 days). 136 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Table 4.3. Average Elapsed Time between Steps in the Procurement Process (days) Borrower first submission Borrower Bank no- to Bank of Bank final no Borrower Borrower bid submission to objection to draft bid objection to draft issue of bid (PreQ) Bank of bid bid Date of (preQ) bid (PreQ) (PreQ) opening evaluation evaluation contract Procurement process steps documents documents documents date/minutes report report signature Issue of Specific Procurement 171.9 169.0 135.1 126.7 226.4 253.0 269.5 Notice Borrower first Submission to Bank 56.6 66.0 121.8 224.3 252.9 286.1 of draft Bid (preQ) Documents Bank final No Objection to draft Bid 24.6 77.7 178.3 199.8 231.8 (PreQ) Documents Borrower issue of Bid (PreQ) 59.7 170.6 190.4 223.3 Documents Borrower Bid (PreQ) Opening 118.4 144.9 178.0 date/Minutes Borrower Submission to Bank of 37.4 73.7 Bid Evaluation Report Bank No-Objection to Bid 48.2 Evaluation Report Source: IEG analysis of a sample of contract data. Note: All contracts in the IEG sample. Multilayered procurement processes and contract approval within client countries can play a significant role in delays after the Bank’s no-objection, to the time the country signs the contract. Therefore, looking at critical shorter intervals, only up to the final “no-objection,” on average, the Bank’s no-objection to the bidding documents is issued 56 days after the borrower submits a first draft. This is more than five times longer than what Bank procedures recommend as a business standard for reviewing or providing comments to bidding documents. 146 Documents presented for review may be returned for revision several times, which affects duration. Although more than 45 percent of the contracts reviewed underwent just a single iteration of draft bidding documents review, another 44 percent required two or more rounds of review, and some (less than 10 percent in IEG’s sample) required three, four, or five iterations. Table 4.4. Average Elapsed Time and Contract Value by Procurement Category Average elapsed time from borrower Number of contracts in dataset first submission to Bank of draft bid Procurement (where contract value is (preQ) documents to contract signature category available) (nos) (days) Average contract value ($) Civil works 137 307.2 15,000,000 Goods 124 287.8 4,709,600 Cons. serv. (QCBS) 99 290.5 2,804,025 Source: IEG analysis of a sample of contract data. Note: QCBS = quality- and cost-based selection. Looking at variations in findings by category, method, and sector, one finding is that procurement processes takes more time, on average, for the selection of consultant 137 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT services using quality and cost-based methods, compared to ICB or NCB, used mainly for goods and works. 147 On average, processing times (to procure consultant services) with quality- and cost-based selection are more than twice as long for contracts awarded using NCB (Table 4.5). Table 4.5. Average Elapsed Time: Issue of Specific Procurement Notice to Contract Signature Procurement method Number of contracts Average elapsed time (days) All contracts 327 270 ICB 155 266 NCB 68 173 QCBS Selection of cons. serv. 82 379 Source: IEG analysis of a sample of contract data. Note: ICB = international competitive bidding; NCB = national competitive bidding; QCBS = quality- and cost-based selection. IEG also analyzed the relative performance of IBRD and IDA countries. 148 Average processing times are longer for contracts in IDA countries, and most of the difference in processing time stems from the very long time taken to procure consultant services (Figure 4.2). 149 Together these results highlight that delays in Bank processes are a reality, and some procurement methods, notably for consultants, are more prone to delays. They also highlight that, as may be expected, country capacity factors are likely a contributing factor. ANALYSIS AND FINDINGS: VARIATION IN ELAPSED TIMES—FREQUENCY DISTRIBUTIONS Average elapsed times do not tell a complete story; the degree of variation around the average describes the proportion of contracts that may take longer to process. For example, although on average 253 days were needed from the borrower’s first submission of draft bidding documents to the Bank’s final no-objection to the final bid evaluation report of the contract, this variable displayed a high dispersion from the average. 150 Half the contracts completed this process in less than 208 days, and the mean time taken is 253, but contracts at the 75th percentile of the distribution take 331 days and contracts at the 95th percentile take 611 days. The five largest processing times in this sample took from 690 to 941 days to process. The degree of variability in processing time also appears to vary by procurement method. National competitive bidding (NCB) contracts show little variability, processing time for both ICB and quality and cost-based selection contracts is widely dispersed, with some contracts taking three years or longer to signature from the issue of the specific procurement notice (Figure 4.3). 151 Overall, the finding that remains true across time intervals is that although most contracts are processed within a reasonable period of time, there is a significant “long tail” of outliers that presumably lead to frequently voiced concerns about delay. One 138 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT likely contributing factor is the degree of complexity in the technical specification of contracts, which is was not possible to proxy because of difficulties of measurement. Delays can also arise from INT processes, outside the procurement anchor. Figure 4.2. IBRD and IDA Countries—Elapsed Procurement Times by Procurement Method (days) 434 IDA 209 342 250 IBRD 173 309 0 100 200 300 400 500 QCBS NCB ICB Source: IEG analysis of a sample of contract data. Note: The time interval measured is from the borrower’s first submission to the Bank of draft bid (prequalification) documents to contract signature. ICB = international competitive bidding; NCB = national competitive bidding; QCBS =quality- and cost-based selection . Figure 4.3. Distribution of Days from Issue of Specific Procurement Notice to Contract Signature All Contracts QCBS 15 60 Number of Contracts Number of Contracts 10 40 5 20 0 0 0 500 1000 1500 2000 0 500 1000 1500 2000 Number of Days Number of Days NCB ICB 30 25 20 Number of Contracts Number of Contracts 20 10 15 10 5 0 0 0 500 1000 1500 2000 0 500 1000 1500 2000 Number of Days Number of Days Source: IEG analysis of a sample of contract data. 139 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT TIME TAKEN TO PROCESS A CONTRACT—FINDINGS FROM REGRESSION ANALYSIS A limitation of the preceding analysis is that although many factors can affect procurement process times, it is difficult to know what the most important determinants of elapsed time may be. IEG therefore attempted a simple multiple regression to look at the relative influence of different factors that affect procurement processing time, including contract attributes (procurement method, category of good or service procured, major sector, and contract value) and country-specific control variables (governance indicators as proxied by the Bank’s CPIA, levels of gross domestic product, and the poverty rate). 152 Two time intervals are examined: The time taken from the issue of the specific procurement notice to contract signature (Appendix Table D.2) and the submission of draft bidding documents to the final no-objection to the bid evaluation report (Appendix Table D.3). One key relationship that emerged from both specifications is that processing time is associated with increased contract value. For every $10 million increase in contract value, processing time increases on average, by 10–14 days. This may be explained by the fact that larger contract values require higher clearance thresholds. Field procurement officers have to request clearance from hub coordinators, regional procurement managers, and the center. The significant correlation between contract value and elapsed times is present even while controlling for regions, sectors, country attributes and gross domestic product. Other variables may also matter, and data limitations may prevent statistically significant results. It is also true that contracts procured using NCB appear to take less time than those using ICB. Finally, governance considerations in the countries concerned may make a statistically significant difference to processing time. 153 To summarize, although only illustrative, the preceding analysis suggests a number of factors that may explain process efficiency and may pave the way for more comprehensive and systematic work in this area. First, average time taken is clearly much longer than Bank norms; second, there is a high level of variability in processing times, typically with a long tail of contracts that take considerably longer than average times. Third, in terms of procurement methods, NCB, even when prior reviewed, is notably quicker than ICB; conversely, consultant contract processing through quality and cost-based methods is particularly time consuming. Fourth, the size of a contract in terms of its value is a prominent determinant of elapsed time. This is likely explained by the implied clearance thresholds. And finally, country capacity and governance matter. Countries with lower CPIAs appear to require longer processing times. 140 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT EFFICIENCY OF PROCUREMENT IN AFRICA – IEG ANALYSIS OF PROCYS DATA Using the Africa Region’s comprehensive database on elapsed time in the procurement process—the only one maintained in the Bank—IEG was able to gain further insights into procurement efficiency, corroborating many of the above messages (Appendix D has full details of the analysis). Elapsed Times in the Procurement Process Each step in a procurement process that involves an exchange between participants in the procurement process is an entry in PROCYS. In 2012, 3,043 requests were logged. 154 On average, there are two to three iterations both between client and task team leader, and team leader and procurement specialist, at each step in the procurement process (Table 4.6). One interesting finding is that in terms of numbers of days of elapsed time per iteration, the client takes the most time with an average— almost 11 days—whereas the task team leader and procurement specialist take around 8 days each. This confirms the importance of country-level factors in terms of determining time taken in procurement. Over time, there is some suggestion of increased efficiency among all parties; the number of days per iteration shrunk considerably for clients, from 12 in 2010 to 5 days in 2012; from 9 to 5 days for task team leaders, and from 8 to 7 days by the procurement specialist (Appendix Table D.4). The analysis also confirms the importance of looking at not just average elapsed time, but its distribution, and confirms that there is a long tail of outlying transactions that take considerably longer than average and that contribute to overall outcomes. The average numbers of days taken by client, task team leader, and procurement specialists was considerably greater than the median of the distribution; for instance, although average days per iteration taken by the procurement specialist per iteration is seven, half the transactions (the median) take fewer than three days. 155 Table 4.6. Africa: Numbers of Iterations and Average Response Times per Procurement Transaction Request, 2009–12 Average numbers of iterations Average number of days taken per iteration Number of Client - TTL—Procurement Client TTL Procurement Year records TTL specialist specialist 2009 1,970 2 2 14 10 8 2010 2,540 3 3 12 9 8 2011 2,778 3 3 12 9 8 2012 3,043 2 2 5 5 7 Avg/Total 10,331 2.5 2.5 10.7 8.2 7.6 Source: IEG analysis of Africa Region data. Note: TTL = task team leader. 141 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT The analysis confirms that clearance at the OPRC level implies long delays in terms of time, with 27 days per iteration. Clearly this partially reflects the increased complexity of such cases. Moreover, the borrower does not always have the capacity to prepare technically adequate documents. Such factors are difficult to reflect in the present analysis. Clients, however, do not take longer to respond to OPRC. There are also clear differences in the numbers of client and task team leader iterations, depending on the clearance level, with an average of five iterations for requests at the OPRC level, four at the regional procurement manager level, and three at the procurement specialist level (Table 4.7 and Appendix Table D.5). Table 4.7. Africa: Response Times for Procurement Categories by Clearance Level, 2009–12 Average numbers of days per Procurement category Clearance level iteration PS RPM OPRC (Number of records) Goods, works, and nonconsulting services 2,690 221 40 Consulting services 4,333 250 4 Client Goods, works, and nonconsulting services 11 16 8 Consulting services 10 13 10 TTL Goods, works, and nonconsulting services 7 7 13 Consulting services 7 10 5 Procurement specialist Goods, works, and nonconsulting services 7 8 16 Consulting services 6 8 25 Source: IEG analysis of Africa Region data. Note: PS = procurement specialist; RPM = regional procurement manager; TTL = task team leader. Delays and Idle Time—Borrower, Bank, and Procurement Staff Besides tracking elapsed times and iterations at each step of the procurement process, the Africa Region database also records “idle” time in between steps. As an example, once bidding documents have been cleared by the Bank, the borrower requires time to prepare for the next step: receiving and evaluating bids, and preparing a bid evaluation report. The procurement office in Africa considers the normal time required for these two steps to be 70 days for consultant contracts and 90 days for goods, works, and nonconsulting services. To the extent that the time in between steps exceeds these norms, it is counted as a borrower delay, which is thus the difference between the “idle time” and the “normal time.” Findings show that borrower delay over and above norms is more important than time taken by the task team. As pointed out by Bank management, multilayered procurement processes in countries also contribute to delays.There are no clear trends over time and considerable variation from one quarter to the next, suggesting that individual transactions may play a large role. 142 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Figure 4.4. Borrower Delays and Bank Response, 2010–12 70 60 50 Number of days 40 30 20 10 0 2010-Q1 Q2 Q3 Q4 2011-Q1 Q2 Q3 Q4 2012-Q1 Q2 Q3 Q4 Borrower Delay Task Team Time Procurement Staff Time Source: Africa procurement office database Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for consultants and 90 days for goods, works, and services). Better tracking systems are just a first step toward making procurement more efficient, in conjunction with other steps. Ideally, tracking systems should integrate potential delays in the Bank that lie outside the procurement anchor, for example, those caused by efforts to control fraud and corruption by INT or the Office of Sanctions and Debarment. There is also scope for increasing efficiency in current Bank processes by streamlining and simplifying the selection process. Current European Union proposals (Appendix A), for example, include shortened time limits for bidding, and IEG found that several client countries have shorter bid times than the Bank requires. With modern communications and more of the process on-line, shortening some periods could be considered. Greater built-in flexibilities—for example, provisions to modify contracts during implementation—could also be considered. The Bank could also consider the introduction of stricter performance penalties for suppliers that have substantially exceeded time limits. Further New Directions for Increasing Efficiency New procurement methods can also offer a means toward increasing efficiency. One example is the use of framework agreements (Box 4.5). Framework agreements can save cost and time, for example, for small but frequent purchases or for repeat phases or purchases that are related to each other. The Bank’s current policy, which introduced the use of framework contracts in 2011, limits their use to common-use goods, simple nonconsulting services, or small-value emergency works contracts, with a contract duration of up to three years and a value not exceeding the NCB limit. In practice, they have been little used under Bank projects, although several 143 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT countries visited confirmed that framework contracts are permissible under local law (Bangladesh, Ethiopia, Mexico, Peru, and Tanzania). 156 Discussions during IEG’s field visits revealed that in Bangladesh, the government developed processes for using framework agreements for the health sector under its own laws. Framework agreements are provided for in Ethiopian law, at the federal and provincial levels. The government of Ethiopia moved forward with its introduction largely unassisted by the Bank, and has entrusted a new agency with responsibility for this—the Procurement and Property Disposal Agency. In Mexico, the Bank facilitated the process of introducing changes into the legal system to provide for the design and formulation of framework agreements. Bank staff played an integral role in establishing a knowledge exchange partnership between the government of Western Australian and the government of Mexico. Box 4.5. New Procurement Methods—Framework Agreements and Reverse Autions Framework agreements, also known as indefinite delivery, indefinite quantity contracts, provide for an indefinite quantity of services during a fixed period of time. These arrangements allow the purchaser more flexibility in both volume and detail of goods or services contracted for. Framework agreements can involve multiple suppliers, allowing the purchaser flexibility to select the best value for each purchase. In a role reversal from a traditional auction, reverse auctions allow bidders to offer the lowest price on the good or service sought by the government purchaser. This is most commonly done on line in real time, creating downward price pressure that often results in a better value to the purchaser. One recent diagnostic report undertaken with the help of the Bank, the Colombia MAPS, discusses reverse auction as well as small-value contracting, shopping, and procurement through commodity markets. Source: IEG. Globally, procurement systems are emphasizing means of improving efficiency that make more intensive use of electronic platforms, and beyond this there is an increasing emphasis on modernization and simplification. The European Union, in its upcoming revisions to its public procurement directive, has proposed several measures to promote and mandate e-procurement—for example, through the use of mandatory electronic notices and communications—a switch to the use of electronic document submission. Also under consideration are proposals to introduce electronic reverse auctions, dynamic purchasing systems, and electronic catalogs. 144 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Though the Bank permits the use of all these systems, it could go further and encourage client countries to move toward their adoption. Box 4.6. E-Procurement There is extensive evidence from the European Union, the Organisation for Economic Co-operation and Development and individual countries that e- procurement has the potential to strengthen governance through its capacity to enhance transparency and improve access to management and audit information. The Aberdeen Group reported (2008) that public sector enterprises have significantly improved their performance as a result of e-procurement initiatives with lower transaction costs, lower incidents of randome and unaccountable or unjustifiable spending, and lower transaction cycle times. These results were echoed in a 2007 World Bank survey of 14 countries, where e-procurement had been introduced, who recounted benefits in terms of transparency, efficiency and market intelligence. Efficiencies for government are also efficiencies for the private sector: the value of e- procurement for suppliers is in terms of efficient market access, ease of bidding, document transfers, business collaboration, efficient transactions, and market information. Potential savings from the successful implementation of e- procurement framework have been estimated by the Organisation for Economic Co- operation and Development to be in the range of five to eight percent of procurement value. When combined with the greater procurement coordination and management control information facilitated by e-procurement, costs have been reported to fall by up to 20 percent. The increase in competition from e- procurement may yield substantial savings through the provision of greater visibility and ease of access to the government market. E-procurement implementation depends on the approach to the development of the technology used and the business process reforms initiated to accompany it. Complex technology and high-risk approaches used in some countries are paralleled with low-risk approaches; incremental development of back office technology- driven functions and tools that have been gradually extended into a fully functional e-procurement market place. The degree of management involvement affects design; some governments have delegated the e-procurement project management role to the software developer, who may have been more prone to dismissing unjustifiable or unsustainable costs. Major sources of concern have been digital signatures being abandoned or corrupted, business models based on limited market information, and poorly understood outsourcing options and risks. Where technology departments have sole leadership, solutions may not correspond to the needs of procurement officials, 145 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT such that these officials have retained paper processes for most procurement activities. Support from MDBs to surmount such obstacles has been weak, based on anecdotal information from MDB partners and IEG interviews. MDBs including the Bank have developed an accreditation model for e-procurement systems to certify whether the system is suitable for Bank projects. However, this accreditation provides very limited help or guidance to countries. The Bank has sometimes provided consultants to countries, raising expectations in relation to e-procurement, but follow-up on how to go about it has been limited, sometimes leading to suboptimal outcomes. There is substantial scope for more support to arrive at better matched solutions. Sources: Aberdeen Group 2008; AfDB, IDB, World Bank, and Curtin University of Technology 2007. BANK INVOLVEMENT IN THE DEVELOPMENT OF E-PROCUREMENT Bank clients have reiterated the need for the Bank to develop or expand e- procurement initiatives. The World Bank Institute (WBI) has delivered structured learning in e-procurement to five countries in the East Asia and Pacific Region using the Global Development Learning Network. WBI has recently converted its e- procurement learning materials into an e-learning course. 157 As part of field visits to select case study countries, IEG sought information about the Bank’s role in e-procurement from country management, Bank procurement staff, and country client counterparts. Both Bank staff and country client counterparts were asked about the extent to which the Bank had focused on new procurement approaches such as e-procurement and whether it had been able to provide support to e-procurement. Bank country management from Azerbaijan, Tanzania, and Turkey indicated that the Bank is not seen as a leader in e-procurement. The Bank plays a minor role in Peru but is not seen as a leader. In Morocco the Bank is seen as a coleader, working alongside (but notably not coordinating well with) the European Commission. Findings are similar in Indonesia, where it was reported that there is no clear view as to the Bank’s leadership role in e-procurement within the development community. Country clients in Bangladesh, Morocco, and the Philippines had favorable opinions of bank support for innovation in e-procurement. Responses from Azerbaijan, Ethiopia, Peru, Senegal, Tanzania, and Turkey did not indicate that the Bank had provided cutting edge support. There were no responses from Indonesia or Mexico. 146 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT It is evident that other development partners have been extensively involved in e- procurement efforts in various field visit countries—in some cases their involvement has exceeded that of the Bank. The message from many of the field visit countries is that the Bank has not been a consistent leader in e-procurement development, and in some cases, Bank involvement has been limited to piecemeal support and ensuring that country e-procurement systems are appropriate for Bank-funded projects. Discussions with staff and clients suggest that to the extent that case study countries already had an existing e-procurement system, Bank involvement focused on certifying whether the systems were appropriate for use with Bank-funded projects. Bank contributions were noted as valuable in: • Bangladesh: The role of the Bank has been substantial and sustained beginning with technical assistance in 2004. Its Public Procurement Reform Project funded e-procurement development and capacity building. The e- procurement program has been especially intensive and has led to Bangladesh rolling out a nationwide system with associated training and business process reengineering; this program is expected to be expanded over the next one to two years. The Bank has also played a role in compliance checking during system development and assisting implementation. Additionally, and independent of project funding, the Bank provided advice regarding the e-signature legislation, which was fully taken on board and led to a re-drafting of the proposed law. • Morocco: The Bank assisted Morocco in the development of a public procurement portal in 2007; this has progressively developed in terms of capabilities and entities that use the system. 158 The portal is now used to publish offers, terms of reference, and results. The European Commission is also supporting Morocco in procurement, especially in training. Both the Bank and the European Union are reportedly acting in parallel and providing complementary assistance, without looking for synergies. • The Philippines: A fairly advanced e-procurement system, PhilGeps was initially supported by the Canadian International Development Agency. It then received additional support from the Bank as well as ADB. After a recent review of PhilGeps, the Bank advocated for the development and implementation of a “virtual store” and electronic reverse auctions. In 2011, the Philippines received global distance learning on e-procurement from WBI. In most countries (including Morocco and the Philippines), the Bank was one among a larger group of international players. It helped build on initiatives originally developed by governments on their own: 147 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT • India: Although not a case study country, reference to India is included due to its major internal efforts at e-procurement, usually at the state or procuring entity level, and with limited homogenization. The Bank has essentially been a follower, providing certification of certain systems for Bank use. • Indonesia: the Bank support included a development policy loan that contributed to procurement reform and capacity building—the latest development policy loan sets targets in the area of e-procurement function. The Bank also undertook an e-procurement assessment as part of a multidonor trust fund initiative. Other development partners have played a larger role in development procurement capacity in Indonesia—AusAID has been the most significant contributor and the Millennium Challenge Corporation is preparing projects and investments that include a technical assessment of the e-procurement system. Bank staff in Indonesia have limited involvement and knowledge of ongoing e-procurement work by other development partners. • Mexico: Mexico was considered an international leader in e-procurement when it began development of the CompraNet system in 1996; the system predates Bank involvement in e-procurement in Mexico. Though the Bank did not have a role in developing CompraNet, it did later provide advice on system improvements so that it was acceptable for use in Bank-financed projects. The Bank’s contribution to e-procurement in Mexico has no clear success record; the Bank has tried to influence the government to shift the focus of the program so that it is procurement driven rather than information technology driven, but has had limited success. • Peru: In Peru, an electronic system—SEACE—has been in place for many years. The Bank, along with the IDB, contributed to its development. SEACE is reportedly problematic due to internal issues, lack of qualified personnel, and an insistence on developing in-house solutions rather than acquiring state-of-the-art technology. Recent Bank support to Peru on e-procurement has been through technical assistance and training staff on Bank procurement, but there has not been an effort to improve the overall system. • Turkey: Development of Turkey’s e-procurement system has been linked to Support for Improvement in Governance and Management, with inputs from the European Union. 159 Bank staff visited Turkey in 2012 to evaluate the system’s compatibility with Bank procurement; so far there has been no substantial Bank participation other than this certification. Finally, case study countries also included some with very limited e-procurement capability. The Bank has in some cases offered support, though the level of engagement varies. • Azerbaijan: In 2005 the Bank attempted to develop an e-procurement training program using an Institutional Development Fund grant, but it did not fare well because of implementation problems that included government 148 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT reluctance about the participation of international consultants. However, the Bank is currently discussing an e-procurement collaboration with the government. The European Bank for Reconstruction and Development recently requested that the government modify its public procurement law in line with the UNCITRAL 2011 model, and the Bank participated in and presented a paper on Bank procurement during the discussions. The Bank is not perceived to be at the forefront of e-procurement in Azerbaijan. • Ethiopia: Ethiopia has a relatively limited e-procurement system that includes online access to bidding documents and manuals, created with the support of the Bank. The Bank has initiated discussions on e- procurement, which led to plans for a feasibility analysis of the current information technology infrastructure in the country. Further action on e-procurement is pending the results of the feasibility analysis. • Senegal: Perhaps the most engaged country, Senegal set up a public procurement website with the Bank’s assistance including procurement plans, tender invitations, requests for proposals, and contract awards. A study is ongoing to determine the feasibility of further e-procurement development, but there are no Bank projects supporting this development. • Tanzania: Tanzania also has very limited e-procurement, allowing for some online submission of procurement information. The Bank’s involvement in Tanzania included advising on a new procurement law, which provides for some e-procurement, feasibility studies, and a planned e-procurement pilot for 2016. Tanzania also received e-procurement training from WBI in 2009. However, questionnaire respondents indicated that the Bank has not adequately provided specific guidelines for e-procurement procedures, nor does it promote the use of e-procurement for Bank procurements, preferring paper methods instead. Use of Institutional Development Funds or Other Grants for E-Procurement Institutional Development Fund or other grants have been used quite frequently, though with mixed success, to support the development of e-procurement in some of the case study countries (see Chapter 2). A fund of $299,000 to strengthen e- government procurement was used in Azerbaijan in 2005 but did not fare well. In contrast, such grants in Morocco were directed to support public procurement reforms, human capacity building, and specifically developing an e-procurement system. A new grant aims to assist Moroccan authorities in disseminating the use of e tools in public procurement procedures. In the Philippines, grants were used to support development of e-procurement. In Indonesia, some of the e-procurement support came from a multidonor trust fund. Other funding instruments have also been used to support e-procurement. In Indonesia there has been e-procurement support through development policy 149 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT lending. Bangladesh had a Technical Assistance Loan that contributed to e- procurement development. Ethiopia referred to an upcoming Specific Investment Loan that intended to include e-procurement, but the idea was abandoned, as it was determined that information technology capacity was too low. 150 APEC (Asian Pacific Economic Cooperation). Bibliography 1999. “Government Procurement Experts Group Non-Binding Principles on Government Procurement”. August Aberdeen Group. 2008. The E-Procurement 1999. Singapore: APEC. Benchmark Report: Driving Year-Over- ASIL (American Society of International Law). Year Superior Performance. Boston: 2013. Will the World Trade Aberdeen Group. Organization Finally Tackle Corruption ADB (Asian Development Bank). 2002. in Public Purchasing? The Revised “Understanding Public Procurement.” Agreement on Government Governance Brief, Asian Development Procurement. Bank. Manila: ADB. http//www.asil.org/insights130415.cf m. ———. 2007. Environmentally Responsible Procurement—A Reference Guide. Manila: Blue, Richard, and John Eriksson. 2011. ADB. “Evaluation of the Implementation of the Paris Declaration: United States AfDB. (African Development Bank). 2011. Government Synthesis Report”. “Evaluation of Paris Declaration Arlington, VA: The United States Implementation at the African Agency for International Development. Development Bank.” Operations Evaluation Department. Tunis: AfDB. Burger, Philippe, and I. Hawkesworth. 2011. How to Attain Value for Money: ———. 2012. “Determination of Financial Comparing PPP and Traditional Thresholds for Modes of Procurement of Infrastructure Public Procurement. OECD Goods, Works and Non-Consulting Journal on Budgeting, Volume 2011/1. Services”. African Development Bank Paris, France: OECD. Group, Procurement Policy and Fiduciary Services Department. January Casartelli, G., E. Wolfstetter. 2007. "Selection and 2012. Tunis: AfDB. Employment of Consultants: the Effectiveness of the World Bank's AfDB, IDB (Inter-American Development Bank), Policy" Working Paper. Washington, World Bank, and Curtin University of DC: World Bank. Technology. 2007. International Survey of Procurement. Perth. Commonwealth of Australia. 2008. Commonwealth Procurement Guidelines, Agaba, Edgar, and Nigel Shipman. 2007l “Public December 2008. Issued under Procurement Reform in Developing Regulation 7 of the Financial Countries: The Uganda Experience.” In Management and Accountability G. Pig and K.V. Thai (eds.) Advancing Regulations, 1997. Canberra: Public Procurement: Practices, Innovation, Commonwealth of Australia. and Knowledge Sharing (Boca Raton, FL: Pr. Academics Press), 373-91. Debevoise, E.W., and C. Yukins. 2010. “Assessing the World Bank’s Proposed Al-Azar, Rima. 2011. “A Tale of Two CDDs: Revision of Its Procurement Lessons Learned for Mitigating the Risk Guidelines,” The Government Contractor, of Fraud and Corruption in CDD 52, (21). Operations.” PowerPoint presentation, World Bank, Washington, DC. Dehn, Jan, Ritva Reinikka, and Jakob Svensson. 2003. “Survey Tools for Assessing Andrews, Matt. 2013. The Limits of Institutional Performance in Service Delivery.” In Reform in Development: Changing Rules Francois Bourguignon and Luiz Pereira for Realistic Solutions. Cambridge: da Silva (eds.), Evaluating the Poverty and Cambridge University Press 128-130. Distributional Impact of Economic Policies. 151 BIBLIOGRAPHY New York: Oxford University Press and ———. 1990. “A Review of Procurement in World Bank. Policy-Based Lending.” Report no. 8341, Washington, DC: World Bank. Dener Cem, Joanna. A Watkins and William. L. Dorotinsky. 2011. “Financial ———. 2002. India Country Assistance Evaluation. Management Information Systems—25 Washington, DC: World Bank Years of World Bank Experience on ———. 2005. The Effectiveness of World Bank What Works and What Doesn’t.” World Support for Community-Based and –Driven Bank, Washington, DC. Development: An OED Evaluation. De Silva, S. 2000. “Community-based Washington, DC: World Bank. Contracting; A Review of Stakeholder ———. 2006a. Debt Relief for the Poorest—An Experience.” Washington DC: World Evaluation Update of the HIPC Initiative. Bank. Washington, DC: World Bank. DfID (Deparment for International ———. 2006b.Engaging with Fragile States: An Development). 2011. “Approach to Evaluation of World Bank Support for Low- Value for Money.” London: Department Income Countries Under Stress. for International Development. Washington, DC: World Bank. European Union, World Bank, and Inter- ———. 2006c. Hazards of Nature, Risks to American Development Bank. 2009. Development – An IEG Evaluation of World Perú Informe de la gestión de las finanzas Bank Assistance for Natural Disasters. públicas Basado en el marco de referencia Washington, DC: World Bank. para la medición del desempeño (PEFA). Lima: European Union, World Bank and ———. 2006d. Small States: Making the Most of Inter-American Development Bank. Development Assistance – A Synthesis of World Bank Evaluation Findings. Gartner Group. 2012. “Survey Shows Why Washington, DC: World Bank. Projects Fail.” http://www.gartner.com/id=2034616. ———. 2007a. The Development Potential of Regional Programs – An Evaluation of GSDRC. 2010. Helpdesk Research Report: Public World Bank Support of Multicountry Financial Management and Frontline Operations. Washington, DC: World Service Bank. Delivery. http://www.gsdrc.org/docs/ open/HD653.pdf,. ———. 2007b. Water Management in Agriculture—Ten Years of World Bank IDB (Inter-American Development Bank). 2010. Assistance, 1994-2004. Washington, DC: “Guide for Acceptance of the Use of World Bank. Country Procurement Systems.” June 2010. Washington DC: IDB. ———. 2008a. Country Financial Accountability Assessments and Country Procurement IAEA (International Atomic Energy Agency). Assessment Reports: How Effective Are 2011. Agency for Financial Regulations World Bank Fiduciary Diagnostics? and Rules AM VI/1: Procurement of Washington, DC: World Bank. Goods and Services, February 16. Mimeo. ———. 2008b. Public Sector Reform: What Works IEG (Independent Evaluation Group). 1977. and Why? An IEG Evaluation of World “The Role and Use of Consultants in Bank Support. Washington, DC: World Bank Group Projects.” Report no. 1824, Bank. Washington, DC: World Bank. ———. 2008c. The World Bank in Indonesia 1999- ———. 1981. “An Interim Report on 2006--Country Assistance Evaluation. Procurement Issues in Bank-Financed Washington, DC: World Bank Projects.” Report no. 3557, Washington, DC: World Bank. 152 BIBLIOGRAPHY ———. 2009a. Bangladesh Country Assistance ———. 2011c. Trust Fund Support for Evaluation 2001-2008. Washington, DC: Development – An Evaluation of the World World Bank Bank’s Trust Fund Portfolio. Washington, DC: World Bank. ———. 2009b. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition ———. 2011d. World Bank Country-Level and Population--An Evaluation of World Engagement on Governance and Bank Group Support since 1997. Anticorruption: An Evaluation of the 2007 Washington, DC: World Bank Strategy & Implementation Plan. Washington, DC: World Bank. ———. 2009c. Review of IDA Internal Controls: An Evaluation of Management’s Assessment ———. 2011e. World Bank Progress in and the IAD Review. Washington, DC: Harmonization and Alignment in Low- World Bank. Income Countries. Washington, DC: World Bank. ———. 2009d. The World Bank Group Guarantee Instruments 1990-2007 –An Independent ———. 2012a. The Matrix System at Work: An Evaluation. Washington, DC: World Evaluation of the World Bank’s Bank Organizational Effectiveness. Washington, DC: World Bank. ———. 2010a. Cambodia – An IEG Country Assistance Evaluation, 1999-2006. ———. 2012b. “The World Bank and Public Washington, DC: World Bank Procurement. An Independent Evaluation.” Approach Paper. ———. 2010b. Climate Change and the World Bank Washington, DC: World Bank. Group. Phase II: The Challenge of Low- Carbon Development. Washington, DC: ———. 2012c. The World Bank Group’s Response to World Bank. the Global Economic Crisis—Phase II. Washington, DC: World Bank. ———. 2010d. IDA Internal Controls: Evaluation of Management’s Remediation Program-- ———. 2013. “Comparison of the International Report on Management’s Implementation Instruments on Public Procurement.” and IAD’s Review of the Five-Point Action Background paper. Washington, DC: Plan. Washington, DC: World Bank. World Bank. ———. 2010d. Mozambique Country Program IMF (International Monetary Fund). 2012. Evaluation. Washington, DC: World Update on the Bank’s Business Bank. Modernization: Results, Openness, and ———. 2010e. Poverty Reduction Support Credits: Accountability Spring 2012.”IMF World An Evaluation of World Bank Support. Bank Development Committee. Washington, DC: World Bank. Kivistö, T., V.M. Virolainen, and E. Tella. 2003. ———. 2010f. The World Bank in Nigeria 1998- Consortia purchasing and logistics in 2007—Country Assistance Evaluation. Kuopio area (Finland)--Lessons learned Washington, DC: World Bank from a 4-year project. IRSPP conference, Budapest, and in KV Thai et al. (eds.), ———. 2011a. “Aid Effectiveness 2005-10: (2005): Benchmarking municipal public Progress in Implementing the Paris procurement activities in Finland. In Declaration.” World Bank, Washington, Challenges in Public Procurement: an DC. International Perspective. ———. 2011b. Capturing Technology for Korpela, J., and M. Tuominen. 1994. Development – An IEG Evaluation of World Benchmarking Logistics Performance with Bank Group Activities in Information and an Application of the Analytical Hierarchy Communication Technologies. Process. Lappeenranta, Finland: Washington, DC: World Bank. Lappeenranta University of Technology. 153 BIBLIOGRAPHY Kruger. K, 2009. “Ban on negotiations in tender ———. 2011c. Strengthening Country Procurement procedures – Undermining the best Systems, prepared for the 4th High Level value for money.” Journal of Public Forum on Aide Effectiveness, Procurement, 4. November 29 – December 1, 2011, Busan, Korea. OECD/DAC Working Mackie, Andrew, and Giovanni Caprio. 2011. Party on Aid Effectiveness Task Force Assessing the Impact of the FEFA on Procurement. Framework. Washington, DC: PEFA Secretariat. OECD-DAC and World Bank. 2004. “Mainstreaming and Strengthening Ministry of Foreign Affairs of Denmark. 2008. Public Procurement: A Strategic “Evaluation of the Implementation of Framework.” Draft report of the 3rd Joint the Paris Declaration: Phase One Round Table on Strengthening Synthesis Report.” Denmark: Ministry Procurement Capacities in Developing of Foreign Affairs. Countries. Washington, DC, and Paris. OECD (Organization for Economic Cooperation Pallas, Christopher L., and Jonathan Wood. and Development). 2003. Mainstreaming 2009. “The World Bank’s Use of the Procurement Function into the Public Expenditure Policy and Effectiveness Country Systems for Procurement: A Dialogue. Paris: OECD. Good Idea Gone Bad?” Development Policy Review 27 (2): 215-230. ———. 2005a. “Paris Declaration on Aid Effectiveness--Ownership, PEFA Secretariat. 2011. Independent Evaluation of the Public Expenditure and Financial Harmonization, Alignment, Results and Accountability Program. Washington, DC: Mutual Accountability.” Paris, France: PEFA. OECD. Pollit, C., and G. Bouckaert. 2004. Public ———. 2005b. “Strengthening Procurement Management Reform: A Comparative Capacities in Developing Countries.” Analysis. New York: Oxford University Volume 3 of the OECD-DAC Guidelines Press. and Reference Series, Harmonising Donor Practices for Effective Aid Delivery. Paris: Sanchez, A. 1997. The World Bank Procurement OECD. Function – Adjusting to Emerging Needs. Washington, DC. World Bank. ———. 2006a. “The Challenge of Capacity Retrieved Development: Working Towards Good from: http://intresources.worldbank.or Practice.” DAC Reference Document, g/INTPROCUREMENT/Resources/Pro Paris. curement-Strategy.pdf. ———. 2006b. “Methodology for Assessment of Singh, J., Dilip R. LImaye, Brian Henderson, and National Procurement Systems.” Paris: Xiaoyu Shi. 2012. “Public Procurement OECD. Of Energy Efficiency Services: Lessons ———. 2009. “Compendium of Country from International Experience.” Examples and Lessons Learned from Washington, DC: World Bank. Applying the Methodology for Tonkin, Christine. 2012. “Procurement Assessment of National Procurement Governance Reform at the International Systems: Sharing Experiences.” OECD Atomic Energy Agency.” Mimeo. Journal on Development 9(4). Thai, K.V., et al. 2005. “Benchmarking Municipal ———. 2011a. International Engagement in Fragile Public Procurement Activities in States: Can’t We Do Better? Paris: OECD. Finland.” In Challenges in Public ———. 2011b. A Practical Guide for Transforming Procurement: An International Perspective. Procurement Systems. Paris: OECD. Academic Press. 154 BIBLIOGRAPHY UNCITRAL (United Nations Commission on ———. 2007a. “Detailed Implementation International Trade Law) Secretariat. Review. India Health Sector. 2006-2007.” (1994, 2011) “UNCITRAL Model Law on INT, Washington, DC: World Bank. Public Procurement.” UNCITRAL, ———. 2007b. “Mandatory Provisions for NCB Vienna, Austria. (for Bank/IDA financed contracts).” Volcker, Paul, Gustavo Gaviria, John Githongo, Washington, DC: World Bank. Ben W. Heineman Jr., Walter Van ———. 2007c. “Operational Approaches and Gerven, and Sir John Vereker. 2007. Financing in Fragile States.” Independent Panel Review of the World Washington, DC. Bank Group Department of Institutional Integrity. Washington, DC: World Bank. ———. 2007d. “Strengthening World Bank Group Engagement on Governance and World Bank. 1997. “The World Bank Anti-Corruption.” Washington, DC: Procurement Function- Adjusting to World Bank. Emerging Needs.” SecM97-854, Washington, DC: World Bank. ———. 2008a. “Enhanced Support to Staff Working in Fragile and Conflict- ———. 2002a. “Fiduciary Management for Affected States.” Washington, DC: Community-Driven Development World Bank. Projects: A Reference Guide.” Washington, DC: World Bank. ———. 2008b. “Use of Country Procurement Systems in Bank-Supported Operations: ———. 2002b. Guidelines for Country Procurement Proposed Piloting Program.” R008- Assessment Reviews. Washington, DC: 0036/5. Washington, DC: World Bank. World Bank. ———. 2009a. “Guidance to Staff for Using the ———. 2003. “Standard Bidding Documents: Company Risk Profile Database Supply and Installation of Information (CRPD).” World Bank, Washington, DC. Systems.” Washington, DC. ———. 2009b. “Handling Suspected and ———. 2004. “Country Procurement Issues Alleged Fraud and Corruption in Paper—Rwanda.” Washington, DC: Procurement: OPCS and INT Protocol.” World Bank. Washington, DC: World Bank. ———. 2005a. “Africa Capacity Building Task ———.2009c. “Piloting Program in Use of Force Report: Forging Effective States Country Systems: First Progress and Engaged Societies.” Washington, Report.” Operations Policy and DC: World Bank. Country Services. Washington, DC: ———.2005b. “Expanding the Use of Country World Bank. Systems in Bank-Supported Operations: ———. 2010a. Financial Management Sector Issues and Proposals.” Operations Strategy FY11-13. 6, 9. Washington, DC: Policy and Country Services, Financial Management Sector Board. Washington, DC. ———. 2010b. “Guidance Note on the ———. 2005c. “Meeting the Challenge of Operational Risk Assessment Africa’s Development: A World Bank Framework (ORAF) – Risks to Group Action Plan.” Africa Region Achieving Results.” Operations Policy Department, Washington, DC: World and Country Services, World Bank, Bank. Washington, DC. ———. 2006. “Quality of Information and ———. 2010c. “Guidance Note: Procurement Communication Technology Arrangements Applicable to Public- Components in Bank Projects.” QAG Private Partnership Contracts Financed Assessment, Washington, DC: World Bnak. 155 BIBLIOGRAPHY under World Bank Projects.” Services under IBRD Loans and IDA Washington, DC: World Bank. Credits and Grants by World Bank Borrowers.” Washington, DC: World ———. 2010d. Monitoring Report 2010. PEFA Bank. Secretariat Report. World Bank, Washington, ———. 2011g. “Strengthening Country DC. http://pefa.org/sites/pefa.org/file Procurement Systems.” Prepared for the s/attachments/ENG-%20MR-10- 4th High Level Forum on Aide FinalReport-May192011.pdf. Effectiveness, November 29 – December 1, Busan, Korea. Washington, DC: ———. 2010e. “Piloting Program in the Use of World Bank. Country Procurement Systems—Second Progress Report.” Washington, DC: ———. 2011h. “Strengthening World Bank World Bank. Engagement on Governance and Anti- Corruption: Procurement Companion ———. 2010f. “Procurement Risk Assessment Piece.” Washington, DC: World Bank. and Management System User Guide.” OPSOR, May 2010, Updated July 2011, ———. 2011i. World Development Report: Conflict, November 2011, January 2012. Security, and Development. Washington, Washington, DC: World Bank. DC: World Bank. ———2010g. “Public Procurement of Energy ———. 2012a. “Analysis of World Bank Efficiency Services: Lessons from Completed Cases of Fraud and International Experience.” Washington, Corruption from the Perspective of DC: World Bank. Procurement.” Integrity Vice Presidency, Washington, DC: World ———. 2010h. “Republic of Moldova Country Bank. Procurement Assessment Report.” Washington, DC: World Bank. ———. 2012b. “Applying the Principles for Good International Engagement in ———. 2011a. “Community-Driven Fragile States and Situations to Development Projects: A Review of Strengthen and Transform Public Governance Issues.” OPCS Document, Procurement Systems.” Background World Bank, Washington, DC. note, OPSOR. Washington, DC: World ———. 2011b. Country Partnership Strategy Bank. Progress Report for the Republic of Moldova ———. 2012c. “The Bank's Procurement Policies for the Period FY09-FY13. Washington, and Procedures: Initiating Discussion DC: World Bank. and Approach Papers--Proposed ———. 2011c. “Diversity Benchmarking Study Review.” Report submitted to the Audit in Selected Countries.” OPSOR, Mimeo. Committee and to CODE, Washington, Washington, DC: World Bank. DC: World Bank. ———. 2011d. “Financial Management ———. 2012d. “Fiduciary Management for Information Systems—25 Years of Community-Driven Development World Bank Experience on What Works Projects: A Reference Guide.” and What Doesn’t.” Washington, DC: Washington, DC: World Bank. World Bank. ———. 2012e. Financial Management and ———. 2011e. “Green Procurement Policy Procurement in World Bank Operations: Framework in Selected Countries.” Annual Report for FY11. Washington, OPSOR, Mimeo. Washington, DC: DC: World Bank. World Bank. ———. 2012f. “Green Procurement in Selected ———. 2011f. “Guidelines: Procurement of Environmental Policy Framework: Goods, Works, and Non-Consulting Review of the World Bank‘s 156 BIBLIOGRAPHY Procurement Policies and Procedures.” ———. 2012p. “Update on the Bank’s Business Background Paper. Washington, DC: Modernization: Results, Openness, and World Bank. Accountability Spring 2012.” DC2012- 0005. Washington, DC: World Bank. ———. 2012g. “Guidance Note for Design and Management of Procurement ———. 2012q. “Virtual VP Review of the Responsibilities in Community-Driven Procurement Policy Review Proposal-- Development Projects.” Washington, An Approach Paper.” Washington, DC: DC: World Bank. World Bank. ———. 2012h. “Investment Lending Reform: ———.2013a. “Approach for Private Sector Modernizing and Consolidating Procurement in Projects financed by Operational Policies and Procedures.” IBRD/IDA, including Public Private OPCS, Washington, DC: World Bank. Partnerships.” Washington, DC: World Bank. ———. 2012i. “OPCPR Procurement Risk Assessment and Management System ———. 2013b. “Audit of the Bank’s Operational User Guide.” Washington, DC: World Framework for Using Investigation Bank. Results in Bank-Funded Projects.” Report No. IBRD FY12-16. Washington, ———. 2012j. “Procurement and PPP DC: World Bank. Transactions Guidance for MDB Public- Sector Engagements and Procurement ———. 2013c. “Procurement in World Bank Principals Applicable to Private Sector Investment Operations. Phase I: A Transactions—Guidance for MDBs.” Proposed New Framework.” OPCS. Washington, DC: World Bank. Washington, DC: World Bank. ———. 2012k. “Procurement Risk Assessment ———. 2013d. “Red Flags Learning Tool.” and Management System User Guide.” Integrity Vice Presidency. Washington, DC: World Bank. http://go.worldbank.org/D1K227ET81. ———. 2012l. “Project Appraisal Document on a ———. 2013e. “Situations of Urgent Need of Proposed Credit in the Amount of SDR Assistance or Capacity Constraints: 26.1 million ($40.0 million Equivalent to Simplified Procurement Procedures and the Republic of Moldova for the Guiding Principles.” Moldova Education Reform Project).” ———. 2013f. “Supporting Countries in Washington, DC: World Bank. Strengthening their Public Procurement ———.2012m. “Public Procurement of Energy Systems: Second Generation Reforms” Efficient Products: Lessons from (draft). Operations Risk Management Around the World.” Energy Sector Department. Washington, DC: World Management Program. Washington, Bank. DC: World Bank. ———.2013g. “Use of Country Procurement ———.2012n. “Review of the World Bank’s Systems.” Background Paper, Procurement Policies and Procedures: Washington, DC, World Bank. Achieving Social Diversity Through the World Bank Institute. 2012. “Public-Private use of Supplier Preferences in the Partnership Reference Guide, Version Procurement Process.” Background 1.0.” Washington, DC: World Bank. paper. Washington, DC: World Bank. World Trade Organization. 1996. “Agreement ———.2012o. “Review of the World Bank’s on Government Procurement (GPA).” Procurement Policies and Procedures, Geneva: World Trade Organization. The Use and Impact of the Bank’s Policy of Domestic Preferences.” Background Paper. Washington, DC: World Bank. 157 Endnotes Chapter 1 1 There are differing views on how and when in the bidding process to qualify firms. The European Union essentially prohibits qualifying firms as part of the bid evaluation process. It must be done solely in advance of the bidding, with no further consideration of qualifications during bid evaluation. In the United States, it is common practice to evaluate qualifications as the final step, after the bid evaluation is completed. In the Bank’s prequalification procedure the qualifications of the bidders are reviewed before the bidders are invited to bid. They are not reviewed as part of the evaluation process, but there is a process (informally used, a post-bid verification) where the borrower confirms that the prequalified applicants continue to meet the qualifications. In the Bank’s postqualification process again, the qualifications are not reviewed as part of bid evaluation. They are reviewed at the end after the lowest evaluated bid price is determined. 2 Some countries have raised another concern regarding bid submission—to permit the use of multiple bid boxes, which some countries maintain can make it easier for bidders to deposit offers in different locations. However, although the Bank could perhaps accommodate different locations for submitting bids, so long it was done in advance of bid opening, and which is not essentially different from submitting bids via courier (allowed), a problem arises when bids are opened at multiple locations. This makes a true public opening impossible. Electronic submission and online opening could eventually take care of this. 3Ideally, participants may need some form of qualification or expertise (possibly from training) and may also need to be accountable. Confidentiality during bid evaluation is important and needs to be protected. 4It is not clear how much the Bank could offer as a silent observer. However, if the Bank does advise the evaluation, questions may be raised as to its neutrality. 5In the Model Law itself, conditions of use for two-stage, request for proposal, and competitive negotiation procedures (which are lumped together), include (a) where it is difficult to formulate specifications, (b) research type contracts, (c) national security concerns, (d) failed tendering, (e) urgent need, and (f) catastrophic event, similar to urgent need. In summary, UNCITRAL seems to say that competitive negotiation is less desirable (higher risk of corruption) and should be used restrictively. However, it is useful (as are two-stage and request for proposal) when technical specifications are not know or difficult to develop. 6 Casartelli and Wolfstetter (2007) raised similar concerns. 7 This appears to be a reference to elapsed time intervals in PROCYS. 8Country ownership was recognized by the Bank as one of the factors critical to successful public procurement reforms (World Bank 2013f). 9 “Regarding the goal of reducing transaction costs, the Bank’s exercise of prior review, under which it verifies compliance with procurement rules, procedures, and policies, is the single most time-consuming element of the oversight process. Prior review would be eliminated in the case of use of country systems” (Pallas and Wood 2009). 10MAPS baseline indicators were used for the first stage of the use of country systems pilot at the Bank. MAPS compliance performance indicators were meant to be utilized as part of Stage III of the piloting program, although no country completed this. 11 Indicators 4(d) and 10(e), respectively. 12 Or, for the minority of countries that included sub-indicator 7(d) in their analysis, 130 out of a possible 165 points. 13 See paras 2.11-2.12 and 2.16-2.18 of the World Bank Guidelines. 159 ENDNOTES 14 Annex C of World Bank (2008b) contains the complete checklist for bidding documents. The checklist was based on sections of the Harmonized Master Procurement Documents, which were approved by the Heads of Procurement Group of the MDBs. 15 Although the overarching indicators and pillars are not traditionally given composite scores as part of the piloting program or MAPS exercises, IEG also calculated average scores received and average number of countries passing for each of these categories. 16 Fifty-seven percent as opposed to 17percent, 18percent, and 44percent for Pillars II, III, and IV, respectively 17 Note that the Turkey team declined to score several subindicators in Stage I, stating that they “need to be addressed at the level of the implementing agency for the pilot project” in Stage III. 18 Subindicator 7(d), “Clarity and transparency of rules for determining whether to engage international or national markets,” is excluded from this list, as this subindicator was only evaluated for three countries, Burkina Faso, Morocco, and Senegal, all of which were found satisfactory in this area. 19 Subindicator 1d. 20 State-owned enterprise participation was restricted for tenders held by that contracting entity, but not for tenders held by other procuring entities. 21 In all, 35 reports were reviewed in 26 countries. The Bank’s Country Procurement Assessment Report included a standard annex checklist for departures from Bank norms in country documents; however, not all reports incorporate this. Some issues listed in project loan/credit agreements are not spelled out in the annex. 22 Bank Guidelines for goods and works include use of country systems as a unique procurement method (although not used so far), although the consultant guidelines do not contain such provisions. 23These results are similar to those noted by Bank management, which find, in a review of 52 countries, that on average there are around 11 NCB provisions per country (World Bank 2013b). 24 The Bank requires at least 30 days or four weeks from the time of advertisement to the bidding deadline. 25 The Mandatory Provisions state only that bidding documents must be “acceptable to the Association.” The Bank Guidelines specify that bid documents may be the national language, but must “provide clear instructions on how bids should be submitted, how prices should be offered, and the place and time for submission of bids.” 26 Negotiation as a method of procurement (direct negotiation, or competitive negotiation) typically takes place at the time of contract award, following a competitive tender process. The Bank’s provisions for NCBs, however, provide that the contract must be awarded to the lowest priced technically responsive bid, with no negotiations permitted. 27 See Appendix A for a survey of areas requiring modification for NCB. 28 Often the status was not specified in the Stage II report and had to be judged by the IEG evaluator. 29 Danish Ministry of Foreign Affairs. 2008. The policies are those related to procurement of goods and works and selection and contracting of consultants, 30 which serve the same function as the Bank Guidelines. 31 “Advanced use” of country systems corresponds to full acceptance of a country’s procurement systems below thresholds established for ICB (as defined in IDB’s Guide for the Acceptance of the Use of Country Procurement Systems, approved by its Board of Executive directors in 2010). However these processes continue to be open to any eligible (national and foreign) competitor. 32 In addition to Turkey, Bhutan was another country that did complete Stage I and II of the use of country systems program, but then chose to follow an alternative path. Bhutan’s small size and the nature of its engagement with donors prompted a decision to first focus on the harmonization of donor and country policies before increasing use of the country system. 33In Azerbaijan, private suppliers appreciated “the good quality of tender specifications” of the Bank’s standard bidding documents. In Morocco, country clients recognized and praised the Bank’s contribution during the design and planning process (exchanges on specifications and evaluation criteria, establishment of procurement 160 ENDNOTES planning). In Tanzania, country clients reported that the requirement of having a procurement plan and Bank support in terms of elaborating comprehensive specifications and terms of reference have helped processing procurement in an efficient manner, meeting required quality and economy objectives. 34 The extent of current Bank involvement in budgetary processes is discussed in Chapter 5 of Volume 1 of this report. 35 Including the World Trade Organization Agreement on Government Procurement, the UNCITRAL model procurement law, and the European Union procurement Directives. 36 The guidelines differentiate between imported goods, domestically manufactured goods with 30 percent or more domestic value added, and other domestically manufactured goods. 37 Proposals for ecolabeling are included in the proposed revisions to the European Union’s public procurement directive (Appendix 7c). 38 This section draws on World Bank (2013b). 39 So far the Bank has not made an effort to follow the ADB in this regard. 40 Value for money is also sometimes elaborated to cover different procurement contexts, such as differences between contracting for works and obtaining professional services. 41 Others apply some value for money principles (such as life-cycle costing, for example) to varying degrees, often without calling it value for money. Some agencies use the term but without defining a distinct procurement process around it—for example, the United Nations procurement guide refers to value for money, but only in limited terms and makes no recommendations to operationalize it in procurement policies and practices. Chapter 2 42 “… ICB is the Bank's preferred approach to procurement. It is intended to offer an efficient and economic approach to procurement, given appropriate circumstances….” The appropriateness of accepted local procedures is recognized in the revised OMS 2.40 of April 1979. 43 IBRD and IDA CDDs were 28 percent of total investment loans approved from FY02 to FY11. The data used were extracted from the Bank’s internal database and the CDD database produced by the Social Development Department. 44 World Bank (2011e), section on Community Participation in Procurement (para. 3.19): “Where, in the interest of project sustainability, or to achieve certain specific social objectives of the project, it is desirable in selected project components to (a) call for the participation of local communities and/or nongovernmental organizations in civil works and the delivery of non-consulting services, or (b) increase the utilization of local know-how, goods, and materials, or (c) employ labor-intensive and other appropriate technologies, the procurement procedures, specifications, and contract packaging shall be suitably adapted to reflect these considerations, provided that these are acceptable to the Bank.” 45 According to the databases maintained by the CDD Anchor in the Social Development Department, of the 2,497 IBRD/IDA investment projects approved during FY02-11, more than a quarter (692) used the CDD approach, at least in part. Development policy operations and additional financing projects and trust-funded CDDs are excluded in the IEG sample. This left 2,044 projects, of which 552 were CDDs. The analysis is supplemented by data from the Bank’s internal database. 46Results are summarized in Appendix D. The CDD anchor further classifies CDD projects into four categories based on their focus/approach: (i) enabling environment for CDDs through policy and institutional reforms; (ii) decision making by participatory, elected local governments; (iii) community control and management of investment funds; and (iv) community control without management of investment funds. A CDD project can have components representing more than one category. For the purposes of this analysis, a project was assigned to the category with the largest allocation in the loan amount 47 In the CDD database, a CDD project can have four different objectives: improving local governance, improving service delivery, community empowerment or improving livelihood security. As most projects aim at more than one objective, for the purposes of this analysis, a project was assigned to the category of the primary objective. 48 Seven projects did not have ISR procurement ratings and were therefore dropped. 161 ENDNOTES 49 In addition, to benefit from synergies with other reviews under this evaluation, projects in “focus countries” and those with higher share of community control and management of investment funds component were given priority in the sample. Focus countries include: (a) field visit Countries (11): Azerbaijan; Bangladesh; Ethiopia; Indonesia; Mexico; Morocco; Peru; the Philippines; Senegal; Tanzania; and Turkey; (b) other focus countries (10): Albania; Bhutan; Colombia; Ghana; Honduras; La; Mozambique; Rwanda; Sierra Leone; and Vietnam; and (c) select countries where INT reviews of CDD projects have been undertaken (6): Cambodia; India, Indonesia; Kenya; the Philippines; and Vietnam. 50 In addition, six countries where INT reviews of CDD projects have been undertaken were also included: Cambodia, India, Indonesia, Kenya, the Philippines, and Vietnam, to enable a closer focus on problem identification. 51 In the Afghanistan National Solidarity project, for example, procurement issues related to procurement under the “district window” component directly managed by the project management unit. Similarly, in Central Africa Republic HIV/AIDS Project, procurement problems related to the centrally managed procurement of the treated bed nets. 52 Implementation of the Indonesia PNPM Rural Project suffered long delays because of difficulties in hiring supervising consultants and facilitators for project coordination and oversight and to provide technical assistance to the communities. 53In the Brazil Maranhao Project, for example, because of the remote location of the project, it was almost impossible to find three suppliers – a problem that the communities tried to overcome by fabricating (with good intentions) three separate bids, thereby opening themselves to charges of fraudulent behavior. 54For example, Bolivia: Second Participatory Rural Investment Project, Brazil: Amapa Sustainable Communities Project, Ethiopia: Pastoral Community Development Project. 55As discussed in country surveys described in Chapter 4, issues of misuse, and fraud and corruption are perceived to be pervasive, and Bank procurement processes within individual projects can only marginally address them. 56This analysis looked at FCS countries from FY06 to the present. In FY06 the Bank used the term low-income countries under stress to describe what is now considered FCS. The LICUS designation was given to those countries with a low CPIA score, and the list was divided into severe, core, and marginal categories. In FY08 a cutoff CPIA score of 3.2 was established and the LICUS list was revised and only included core and marginal categories. In FY09 the list was entitled “Fragile States List” and the cutoff score was revised to 3.0 for core fragile states and 3.2 for marginal. In FY11, per an agreement at the beginning of IDA 15 and along with other multilateral development banks, the World Bank has defined “fragile situations” as having either harmonized average CPIA rating of 3.2 or less or the presence of a United Nations and/or regional peacekeeping or peace building mission (for example, AU, European Union, OAS, NATO), with the exclusion of border monitoring operations, during the past three years. Most recently, in its new operational directives for investment lending (OP 10.0), expected to be in force from April 2013, the Bank has expanded its definition of fragile situations to include cases where the borrower/beneficiary is deemed by the Bank to experience capacity constraints because of fragility or specific vulnerabilities (including for small states). 57 Multidonor trust funds are typically recipient-executed trust funds. These findings are in line with the findings of the Scanteam (2010) report, which highlighted the increase in use of MDTFs as they become the preferred source of funds for operations in FCS, with IDA resources used less frequently. 58 The10 FCS countries were selected in a way as to include countries from different regions which have regularly appeared on the FCS country list. Countries selected were Afghanistan, Bosnia and Herzegovina, Burundi, Haiti, Lao PDR, Liberia, Nepal, Sierra Leone, Timor-Leste, and West Bank and Gaza. 59 The same countries were selected for the project level analysis as for the CAS review: Afghanistan, Bosnia and Herzegovina, Burundi, Haiti, Lao PDR, Liberia, Nepal, Sierra Leone, Timor-Leste, and West Bank and Gaza. 60 Emergency procedures are selectively applied, with country department clearance, to those operations that appear to merit them, and not to all operations in an FCS. There is no flag in the Bank internal database for OP 8.0 projects. IEG therefore used “emergency projects” as a proxy, under the assumption there is a higher likelihood of OP 8.0 being used in these projects. 162 ENDNOTES 61The Bank has, however, created some incentives designed to encourage and facilitate staff placement in FCS (see World Bank 2008a.) 62 As a result, frequently there was a loss of time with the use of ICB and international advertising, because of little external participation. 63 Excluded from the review is the procurement of off-the-shelf equipment and packaged software, such as printers, desktop computers, and similar devices. Also excluded are advisory projects that assist with policy or market development, such as telecommunications connectivity or e-governance. 64It was challenging to construct a sample, since information technology components are embedded within a range of sectors. IEG was able to develop a sample of projects based on the following criteria: (i) identified in the Bank project database as having ICT components; (ii) cited by the task team leader as ICT being critical to achievement of development outcomes; (iii) loan amount exceeding $50 million; and (iv) approved within the last 10 years, but in implementation for at least two years. The sample was adjusted to include additional projects recommended by procurement specialists See World Bank (2003). There are two versions of the documents to cover two-stage and single-stage bidding. 65 These documents were last updated in 2008. 66 The revisions are reflected in clauses 2.6, 2.20, and 3.6 of the guidelines. 67 Clause 2.6 of the guidelines. 68Clause 2.20 of the guidelines addresses the issue of technical specifications that might refer to specific brands or manufacturers. 69 Clause 3.6 of the guidelines, added through the 2011 revisions. Note that framework agreements could also be useful in many other areas of small and repeat purchases, for example, in education or health projects. IEG field surveys found that the Bank's current policy is somewhat limited (to use in emergencies or small works, and for a duration of no more than five years). On average, country clients perceived Bank advice in this respect as being modest. Several country visits confirmed that framework contracts are permissible under local law (Mexico, Peru, and Tanzania) but had not been acceptable to the Bank. In Bangladesh, the government has developed processes for using framework agreements under its own laws and uses them in its health sector. 70 The majority of respondents/interlocutors were project procurement staff, though some task team leaders also participated. 71 This contrasts with findings of the IEG ICT evaluation (IEG 2011b). 72 Prequalification is interpreted in different ways in different jurisdictions. The European Union Court of Justice, for example, required the separation of “selection criteria” and “award criteria,” with prequalification used in the former. In the United States, however, past performance is reviewed as an element of award criteria. 73 Procurement for the Bangladesh Central Bank Strengthening Project had to be restructured after an initial failure of bidding. What is reported here is how well the restructured procurement proceeded. It too encountered delays with one package, but overall was rated successful. 74 In one case, seven firms obtained bidding documents but only two bid. In this case, there did not seem to be an issue with the specification of goods to be procured thought there was some suggestion of possible collusive practices. 75 Similar views were expressed by Bank country staff in Turkey, who were especially supportive of competitive negotiation as a method of procurement. 76 This recommendation is in line with management’s response to IEG’s ICT evaluation. 77 This recommendation is in line with IEG’s ICT evaluation, which recommends looking accelerating clearances. 78Alsoraised in IEG’s ICT evaluation. In its response management offered to address the procurement capacity constraint through training and other means. To determine the World Bank PPP portfolio, IEG applied definitions used by the Bank’s Get PPP anchor, 79 which had undertaken a detailed scrutiny of the infrastructure PPP portfolio over the years FY10 and FY12. 163 ENDNOTES Under the guidance of Get PPP, and in line with an ongoing parallel IEG evaluation of PPP, IEG then constructed a selection of PPP projects for FY02-12 across all sectors. The selection was based on searches of PADs. 80 IEG selected the 46 sample PPP projects from a list of 73 projects based on data from FPD and Get PPP. Of the 73 projects, 27 related to creating an enabling environment and pipeline development and were not reviewed. The remaining 46, which involved Bank investment, were then reviewed. 81 Background Bank information can be found through the PPP in Infrastructure Resource Center: http://PPP.worldbank.org. This site provides a number of PPP resources and links to more, and also contains a database of PPP projects. 82IEG interviews with Bank procurement staff showed a positive reception to the new provisions in terms of constraints on PPP projects. However, there appears to be differing interpretations of the revisions, especially reference to “procedures acceptable to the Bank.” Some read this as a greater reliance on the principles stated in section I of the Guidelines (economy, efficiency, fairness, transparency). Others believe it invokes adherence to section II, the rules for ICB, which are much more prescriptive. Overall, though, the revisions are seen to bring the Guidelines closer in line with the principles of international good practice for PPP: compliance with local law; fairness, transparency, and competitive processes; and concession contracts that are reasonable in terms of price, quality, and risk allocation. 83An overall portfolio of 73 PPP projects was assembled by combining two PPP project databases, one from FPD and the other from Get PPP, comprising 55 Get PPP projects (FY10-12) and 18 FPD projects (FY01-12). The list of projects created under the guidance of and covered the period from the beginning of FY10 up to December 2011. 84The large number of projects from Africa Region may be because financing needs in this region are most acute, and hence there is a bigger role for the private sector. 85 One was a partial credit guarantee, and two were grant-funded under a carbon finance facility. the Bank provided a guarantee in two projects—the Botswana-Morupule B Generation Project 86 Additionally and the West Africa Economic and Monetary Union Capital Markets Development Project. 87 IFC’s equity investment was provided to the Indonesia Infrastructure Finance Facility. Chapter 3 88 Procurement risk involves both process risk (rules and procedures will not be followed) as well as outcome risk (procurement actions will not lead to optimal outcomes in economy). Present Bank practices emphasize process risk, which is intended to support underlying Bank procurement principles of transparency and competition and to help contain fraud and corruption. 89IAD Report on an Audit of the Management of Procurement Risks for Bank-Funded Projects July 2012. 90 On March 30, 2012, proposed revisions were adopted by the World Trade Organization to its Government Procurement Agreement that would explicitly require agreement parties to fight corruption in public procurement. If the proposed revisions are accepted, it will be the first time that a legal instrument of the World Trade Organization has directly addressed the issue of corruption in international trade regulation through the public procurement systems of member countries. See ASIL (2013). 91ICB contracts accounted for around half in terms of numbers of all prior review contracts in FY12 and FY11, based on IEG’s analysis of the Form 384 database. 92 Except in the Middle East and North Africa Region which has maxium thresholds per country and not per project. 93 See Appendix C on the Bank’s methods thresholds. Other MDBs/IFIs have oversight processes that have some differences compared to the Bank, although their guidelines are very similar. For example, at the IDB, all contract approvals are undertaken by field-based staff. Conversely, AfDB clears contracts through a single procurement unit in headquarters. Its limited field staff (of around four people) also report to Manila. With a much smaller procurement operation, much of the oversight is done by sector departments. ICB and prior review thresholds are generally the same, and the procurement unit is only involved with ICB contracts. 164 ENDNOTES Contemplated changes under discussion would put more oversight responsibility to the sectors, coupled with an accreditation program for sector staff. 94 In shopping, formal advertisement and invitations to receive bids are not required but rather several price quotations from at least three suppliers or contractors are compared to assure competitive prices. Other types of procurement methods for goods and works include: Limited International Bidding, Direct contracting, Contracting without competition (single source); and Force account. From 2011, framework agreements – long- term agreements that set out terms and conditions under which procurements are made throughout the term of the agreement—were added to the list. See World Bank (2011e). 95Prior review thresholds have been set as early as the 1980s. The first documented prior review thresholds that IEG located come from 1992. Prior review thresholds were set at the regional level (Sanchez 1997). 96IEG was able to find historical prior review and method thresholds for the following regions: Africa: 2004 (selected countries), 2005 (selected countries), 2010, 2012; East Asia and Pacific: 2006, 2011, 2012; Europe and Central Asia: 2006, 2007 (four revisions), 2010, 2011; Latin America and the Caribbean: 2007 (Colombia, Mexico, Brazil only), 2011; and South Asia: 2008, 2012. 97 BP11.0 was updated in January 2011 and revised again in April 2013. The 2011 BP 11.0 includes the following annexes: Annex A - General Responsibilities and Accountabilities for Procurement Work; Annex B - Decision Authority Matrix; Annex C - Maximum Prior Review Thresholds; Annex D - Mandatory Prior Review Thresholds for RPMs and the OPRC; Annex E - Handling of Procurement Complaints on Contracts Financed by the Bank. There are no changes in the cited thresholds. 98 The Bank’s Procurement Policies and Procedures: Initiating Discussion and Approach Papers – Proposed Review, February 29, 2012. 99 Examples were given of suppliers in the power sector and construction sectors that met all technical specifications and had the lowest price but where clients still perceived qualitative differences. 100 http://www.transparency.org/research/cpi/. In addition, two Kauffmann and Kraay Governance indicators were used for this analysis: Government Effectiveness and Control of Corruption. 101The overall CPIA score and two sub indicators were used for this analysis: sub indicator 13 Quality of Budget & Financial Management and sub indicator 16 Transparency, Accountability & Corruption in Public Sector. 102Additional results are presented in Appendix 5a for subgroups of countries and under alternative assumptions. Correlations presented here represent the most favorable scenarios; lower correlations are obtained under alternative scenarios. 103However, it has also been pointed out that not all completed Post-Procurement Reviews are uploaded into the Bank’s electronic Post-Procurement Review system, in the post-review module of P-RAMS, as required by the Bank’s Guidance Note to Bank Staff to Conduct Post-Procurement Reviews and IPRs (World Bank 2010c). Other issues have been raised, for example, the independence of such reports, and the extent of their 104 availability. Although Post-Procurement Reviews are ordered by Bank procurement staff, they are sometimes undertaken by the same persons, because of resource issues. Consultants used on Bank operations may also be used for Post-Procurement Review s, and they may sometimes undertake several series of Post- Procurement Reviews. Such practices could lead to conflict of interest. 105 FY 09 Annual Procurement Report, OPCS, World Bank. For example, the FY11 annual report makes almost no reference to the substantive findings of the Post- 106 Procurement Reviews, except to say “Post-review helps to strengthen client capacity through the actions taken to address identified issues.” 107 Bangladesh: Post-Procurement Review FY11 consolidated report. 108The sample consisted of 69 PADs from projects approved 2002 to June 30, 2012, based largely on the overall IEG country selection framework described in Chapter 1, drawing on projects in each region and in a combination of sectors. In all, there were 19 countries in the sample as follows: Azerbaijan, Bangladesh, Brazil, Cambodia, China, Ethiopia, Ghana, India, Indonesia, Lao PDR, Mexico, Morocco, Mozambique, the Philippines, 165 ENDNOTES Rwanda, Senegal, Tanzania, Turkey, and Vietnam. The sample was further divided between projects approved before and after the introduction of P-RAMs (in mid 2010). The sample contained 39 pre-P-RAMs projects and 30 projects approved post P-RAMs. 109 Project staff (including the task team leader, the project-relevant Bank procurement specialists, and representatives of the borrower and its implementing agency have a wide array of instruments with which to assess and manage these risks (summarized in Appendix 3a, Box 4) and the evidence shows that they exert significant flexibility within the overall procurement policy framework set by the Bank’s central guidelines. The procurement plan is essentially a team effort. 110These markers are only indicative empirical measures; risk items are not equal in themselves. Moreover, PADs with a greater number of identified risks were often also more highly articulated in their overall treatment of risk; those with fewer identified risks were often more cryptic in their treatment of risk issues, more vague about risk ratings, and less specific in their risk mitigation plans. 111 Disaggregating and classifying, the following categories emerge: Country Procurement Rules and Oversight (local law and public procurement agencies); Project Design issues (most generally decentralized procurement); and Implementation Capacity (comprised of procurement documentation and record keeping and staffing issues). There were also issues relating to the Procurement Preparation Phase (advertising, bidding documents); the Evaluation and Award Phase (including issues relating to the Bid Evaluation Committee; and complaints issues); Fraud and Corruption (collusion; fraudulent documents; contract packaging issues); and Contract Management (downstream risk of fraudulent documentation; quality of construction). Most came in projects approved after 2007, following implementation of the governance and anti-corruption 112 program, as well as publication of the Detailed Implementation Review reports completed by INT on the India Health Sector. 113Although this may reflect diminished numbers of CPARs in the latter period, other diagnostic work could also have been referred to in principle. 114The Procurement Risk Assessment Questionnaire contains a checklist of 11 risk factors, and 60 subquestions (Box 3.4). Once the procurement staff enters risk rating into the P-RAMS template for each risk factor, an overall procurement control risk rating (which P-RAMS labels “Project Risk”) is assigned to the project. The Mitigation Measure Action Plan describes risks and assigns their mitigation to a responsible entity with a planned due date and completion date. 115The questions in the assessment are not mandatory or exclusive, but are intended to aid the procurement staff in the risk assessment process by focusing on the types of generic risks that are likely to be faced when assessing any implementing agency in a given project. It was an explicit OPSOR intention to have only a “soft integration” of P-RAMS and ORAF (hard integration 116 would be too data heavy). 117There is significant variance in compliance rates between regions. East Asia and Pacific, Europe and Central Asia, and South Asia were above the Bank-wide average of 72 percent (80 percent, 81 percent and 72 percent respectively) whereas the Middle East and South Asia (61 percent) and Africa (58 percent) were below average in their completions. 118 Quantitative sections of the questionnaire gave P-RAMS an average score of 2.6, across respondents, that is, somewhere in between a modest and substantial risk mitigation tool. 119Risk Factor 4 (Staffing—the code of ethics for staff); Risk Factor 5 (Procurement Planning—collusion risk); and Risk Factor 8 (Evaluation and Award of Contract—unusual bid patterns); and Risk Factor 11(Procurement Oversight on anti-corruption agencies and their role in independent oversight). This refers to the following questions: Risk Factor 5 question 3; Risk Factor 8 question 3; and Risk Factor 10 120 question 6. 121For FY10, there were 80 instances of misprocurement, of 22,717 contracts awarded, and for FY11, there were 52 instances, of 17,713 contracts awarded. The Bank’s complaints database maintained by the procurement anchor is distinct from INT’s database on 122 complaints. The former can be triggered by any complaint related to the procurement process, and some may 166 ENDNOTES have (or may lead to fraud and corruption aspect. INT’s complaints refer to (suspected) cases of fraud and corruption. 123 According to management, prior review contracts account for around a fifth of the Bank’s total annual contracts, which amount to some 100,000 contracts per year. The most frequent causes for complaints in recent years have been: technical specification issues contained in the requests for proposals; alleged irregularities in either or both technical and financial evaluation of proposals; and awards of contract, often citing lack of qualification of winning vendors, sometimes accompanied by allegations of fraud and corruption. Cases of the latter may lead to INT referrals and entry into a separate INT database, which may lead to further investigation. The number of referrals that have resulted in INT mounting an investigation have been in a range of 90 –150 124 per year. 125INT is equipped to address fraud and corruption issues at two broad levels: investigation and prevention. Historically INT’s main role was to conduct investigations, which were initiated by request of the operating regions, or on its own account. The main vehicle for such investigations was the Detailed Implementation Review, which consisted of in depth analysis and data gathering on a large sample of contracts in several projects. They brought forth abundant information on fraud and corruption. Since discontinued, they have been replaced by individual contract investigations and final investigation reports, which are triggered by a combination of referrals from the operating regions, and/or from other sources, including from a hotline available to the public. 126Its 10 principal flags are complaints from bidder; multiple contracts below procurement threshold; unusual bid patterns; seemingly inflated agent fee; suspicious bidder; lowest bidder not selected; repeat awards to same contractor; changes in contract terms and value; multiple contract change orders; and poor quality of works and services. 127Undisclosed INT working paper provided to IEG: “Risks and Lessons relating to Implementation Identified through FY09-12 Final Investigation Reports and Internal Investigations.” 128 Under Bank rules, collusion is deemed to be a part of fraud and corruption, though in some jurisdictions (for example, the European Union or the United States) they are legally separate issues. 129 Many complaints did not lend themselves to easy classification. Those identified as being procurement related but whose details did not specify a particular procurement stage, were deemed to be “general procurement complaints.” Financial management complaints have been classified as such, and some which had to do with issues in implementation (for example, displacement of people as a result of project activities) have been classified accordingly. A large number could not be linked to any stage in the project cycle (corruption, bribery, nepotism, and so forth), and these have been classified as “other/to be determined.” 130There is also an anti-money-laundering list maintained by the Procurement anchor at the Bank that staff are required to consult. 131An analysis of the incidence of sanctions by Bank management observed that from 2007 to 2012, 157 contracts, concerning 54 projects in 33 countries had led to sanctions. There is no discernible pattern by category or method though there appears to be some association with the transport sector, small civil works, and countries with low CPIA indicators. 132 If OSD determines that there is sufficient evidence to support INT's accusations (that is, that it is “more likely than not” that sanctionable misconduct has occurred), Office of Suspension and Debarment issues the case to the firm and temporarily suspends the firm from eligibility for new Bank-financed contracts pending the final outcome of the sanctions proceedings. The firm has 30 days to submit an “explanation” to the Office, on the basis of which the case can be withdrawn or the recommended sanction revised, and 90 days to appeal the case to the Sanctions Board. If the firm does not appeal to the Sanctions Board, the case ends at the first tier, and the sanction recommended by Office of Suspension and Debarment (usually a form of debarment) is imposed, superseding the temporary suspension. To date, 60 percent of sanctions proceedings have been resolved at the first tier (that is, without an appeal from the firm). If the firm does appeal to the Sanctions Board, there is a de novo review of INT's case, with the Sanctions Board making a final determination as to whether there is sufficient evidence to support INT's accusations and, if so, what sanction is appropriate. 167 ENDNOTES 133The respondent may contest this action and the Evaluation and Suspension Officer can lift the temporary suspension. 134 According to INT, the Company Risk Profile Database will be enhanced in FY14, to include a more robust search engine along with being integrated into the electronic no-objection system. 135Task team leaders in the Roads Sector point to difficulties in getting good contractors who respect Environment, Health and Safety considerations. However, the Bank does not require ISO 18001, 14001, or other certificate. Requests were made for the Bank to develop a “performance database” to track both negative and positive information. 136 The Protocol for Handling Suspected and Alleged Fraud and Corruption in Procurement establishes a process for how to handle this “grey zone,” as does the Guidance Note for Staff for Using the Company Risk Profile Database. See World Bank (2009a, 2009b). Chapter 4 137 Including, ultimately, the use of computer-based forensics for review of bidding documents. 138Form 384 user guide from April 2009 mentions that its creation dates from 1998. Successive versions of the form and its progressive simplifications are discussed in Appendix 4b. http://intresources.worldbank.org/INTPROCUREMENT/Resources/384-Users_Guide.doc 139 In July 2008, the World Bank and the IDB signed a Memorandum of Understanding to share use of this tool. 140Available at www.iniciativasepa.org . SEPA was developed by the Bank, starting in 2006. In 2008, the World Bank and IDB signed a memorandum of understanding "to share use of this tool and thus improve governance of projects and contribute to the harmonization of processes contributing to achieve the Millennium Development Goals." 141Countries using SEPA (as of January 2013): Argentina, Armenia, Bosnia and Herzegovina, Colombia, Costa Rica, El Salvador, Guatemala, Honduras, Mexico, Moldova, Nicaragua, Panama, Peru, Dominican Republic, Uruguay. Belarus and Ukraine are also mentioned as users of SEPA on the SEPA website, however Bank management note that this is not the case. SEPA is being used for a single World Bank funded project in Chile. Work on a new version is ongoing with a main focus of SEPA 3.0 being the connectivity with other systems. 142Its effectiveness depends on the visibility of the SEPA platform to civil society and private firms. Users of the public website, in principle, can filter for countries, project, procurement method, and category to access the latest procurement plans of regional or thematic areas they are interested in including information on estimated and actual contract amounts, contract sign and completion dates, and details on bidders. 143Operations Procurement Electronic No Objection System (http://intranet.worldbank.org/WBSITE/INTRANET/OPERATIONS/INTPROCUREMENT/0,,contentMDK:2 3308187~menuPK:8955261~pagePK:60000209~piPK:60000211~theSitePK:278020,00.html). 144 Appendix B has details. Although IEG drew up a specific list of prior review contracts per country to construct a representative stratified sample, the response rate from country offices was highly variable in quantity and quality. Each regional office has its own format for maintaining contract data (including as may be expected, different currency units and time recording conventions, as well as its own unique reference number, which is not consistently linked to the Bank-wide Form 384. Clearly, this affects reliability of the results obtained and will have to be a caveat to their interpretation. 145 IEG also looked at a shorter interval, from the borrower’s first submission of draft bidding documents to the Bank’s Final No Objection to the Bid Evaluation Report of the contract. This interval is likely to be more meaningful for two reasons. First, Issue of Specific Notice in practice may occur before or after work on draft bidding documents begins. Second, ending with the Final No Objection better reflects steps within the procurement process itself, in contrast to country level factors that may affect contract signature. On average, 253 days were needed, that is, around 30 days more are needed for going to contract signature, which took 286 days. 146The business standards established in Bank Procedure BP 11.00 – Procurement (http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/EXTPOLICIES/EXTOPMANUAL/0,,contentM DK:20064756~menuPK:4564187~pagePK:64709096~piPK:64709108~theSitePK:502184,00.html) address 168 ENDNOTES specifically the response time needed by the procurement team; time taken by other parts of the Bank’s task team are not included. This analysis does not distinguish between different parts of the Bank’s project teams; results presented are for the overall time taken by the Bank. 147From the time the borrower issues the Specific Procurement Notice to signature of the contract. Further detailed breakdowns are in Appendix A. Based on the field visit sample in the IEG dataset. IDA countries are Bangladesh, Ethiopia, Senegal, and 148 Tanzania; IBRD countries are Azerbaijan, Indonesia, Mexico, Morocco, Peru, the Philippines, and Turkey. 149 Using quality- and cost-based selection. 150Though the mean time in the sample is 253 days, the standard deviation is as high as 160 around the mean value. 151 Similar results are obtained if a narrower time interval, only up to the final no-objection, is examined. 152One caveat to these findings is that linear regressions are based on the assumption of underlying normal distributions, which may not apply for some of the variables here. 153This is the case in one regression specification, which is focused on only the procurement-related steps of the interval. 154 2009 was a startup year and is therefore not used as a baseline. 155The database shows some errors in recording times. For example, in 62 cases (of 10,331) the average number of days that client take to respond has a negative values. However, these are not frequent. 156As discussed in Chapter 1 of this volume, proposed revisions to the European Union Procurement Directive go beyond this, to recommend the introduction of the “innovative partnership” procedure, which allows procuring agents to enter into partnership arrangements with suppliers to develop and subsequently supply new and innovative products, works, and services according to specified requirements that are performance oriented. 157From the WBI e-procurement online course: The objective of this learning course is to support MDB member countries to effectively adopt e-government procurement by providing tools that help place it within a broader country procurement reform. This course consists of a conceptual framework of e-government procurement aimed at helping participants develop a common understanding of fundamentals and components that can be used to supplement the existing MDB reform dialogue on procurement. 158 The Bank provided specific assistance in the following: development of a multi-criteria nomenclature aiming at facilitating research of calls for bids by types of services (works, supplies, and services); developing a monitoring system in view of alerting agencies on failures to comply with publishing requirements, developing multimedia guides for public buyers and bidders. 159 Revisions to European Union directives proposed in 2011 place substantial emphasis on e-procurement and propose to make its use mandatory, based on the belief that e-procurement is an important tool to assist small and medium-size enterprise development by making procurement opportunities more accessible and lower costs to bid. 169