95605 Road Management Policy An Approach to the Evaluation of Road Agency Performance Michael Ian Pinard Working Paper No. 105 Road Management Policy An Approach to the Evaluation of Road Agency Performance Road Management Policy An Approach to the Evaluation of Road Agency Performance Michael Ian Pinard April 2015 The SSATP is an international partnership to facilitate policy development and related capacity building in the transport sector in Africa. Sound policies lead to safe, reliable, and cost-effective transport, freeing people to lift them- selves out of poverty and helping countries to compete internationally. ******* The SSATP is a partnership of 40 African countries 8 Regional Economic Communities 2 African institutions: UNECA, AU/NEPAD Financing partners for the Second Development Plan: European Commission (main do- nor), Austria, France, Norway, Sweden, United Kingdom, Islamic Development Bank, Afri- can Development Bank, and World Bank (host) Many public and private national and regional organizations ******* The SSATP gratefully acknowledges the contributions and support of member countries and its partners. ******* This paper is a product of the SSATP, written by an external author. The findings, interpretations, and conclusions expressed herein do not necessarily reflect the views of the SSATP or the World Bank. The SSATP does not guarantee the accuracy of the data included in this work. The boundaries, colors, de- nominations, and other information shown on any map in this work do not imply any judgment on the part of the SSATP or the World Bank concerning the legal status of any territory or the endorse- ment or acceptance of such boundaries. © March 2014 The International Bank for Reconstruction and Development / The World Bank Group 1818 H Street, NW Washington D.C 20433 USA. The text of this publication may be reproduced in whole or in part and in any form for educational or nonprofit issues, without special permission provided acknowledgement of the source is made. Re- quests for permission to reproduce portions for resale or commercial purposes should be sent to the SSATP Program Manager at the address above. The SSATP encourages dissemination of its work and normally gives permission promptly. The Program Manager would appreciate receiving a copy of the document that uses this publication for its source sent in care of the address above. Photo credit: iStock photo - Roadworks on the motorway out of Cape Town, South Africa. Contents Foreword ________________________________________________________ vii Acknowledgements ________________________________________________ xi Abbreviations and Acronyms _______________________________________xiii Executive Summary _______________________________________________ xv 1. Introduction ________________________________________________ 1 Context _____________________________________________________ 1 Objectives ___________________________________________________ 5 Structure ____________________________________________________ 5 2. A Holistic Approach to Assessing Road Agency Performance _________ 7 Challenges faced by Road Agencies in Africa ________________________ 7 Factors Affecting Road Agency Performance________________________ 8 Road Asset Management ______________________________________ 10 3. Assessment of Road Agency Performance in Asset Management _____ 13 Introduction ________________________________________________ 13 The CRM Framework for Assessing Road Agency Performance in Asset Management ________________________________________________ 14 4. Outcomes of Alternative Approaches for Assessing Road Agency Performance in Asset Management _____________________________ 25 Introduction ________________________________________________ 25 Outcomes from Application of CRM and PAS Frameworks___________ 26 Scope for Improvement of CRM Framework ______________________ 31 Need for Performance Indicators ________________________________ 32 v Road Management Policy– Road Agency Performance 5. Performance Measures and Indicators __________________________ 33 Introduction ________________________________________________ 33 Existing Performance Indicators ________________________________ 34 Proposed Performance Indicators _______________________________ 36 Harmonization of Performance Indicators ________________________ 39 Integrating Performance Measures and Indicators into a Road Agency __ 40 6. Findings, Conclusions and Recommendations ____________________ 43 Main Findings and Conclusions ________________________________ 43 Annexes _________________________________________________________ 47 Annex A. Outcome of assessment of road asset management capacity in South Africa based on CRM and PAS evaluation frameworks _________ 47 Annex B. Outcome of assessment of road asset management capacity in Tanzania based on CRM and PAS evaluation frameworks ____________ 49 Annex C. Outcome of assessment of road asset management capacity in Botswana based on CRM and PAS evaluation frameworks____________ 51 References and General Bibliography _________________________________ 53 vi Foreword Efficient and effective road transport is central to the economic growth and develop- ment of all African countries, this mode accounting for about eighty to ninety percent of the continent’s total trade in goods and services. For this reason countries need adequate road infrastructure management policies, strategies and institutions to man- age this crucial asset in an optimal manner including systematic means of measuring the performance of the road agencies as a basis for determining those factors that aid or impede the attainment of their desired results. In an endeavor to deepen the understanding of the factors that affect the performance of road agencies in road asset management, two evaluations of road agency perfor- mance using two different approaches have been conducted in Africa during the past five years. The first approach is the Commercialized Road Management (CRM) framework of assessment explained in the SSATP Working Paper No. 92. In this study, the criteria were defined so as to allow for quantitative measurement and com- parison of road sector performance across countries. The CRM approach is a broad approach of performance evaluation in that it focuses on the extent of an agency’s compliance with the requirements of the four Building Blocks of the Road Manage- ment Initiative (RMI) pertaining to Responsibility, Ownership, Financing and Man- agement that are believed to critically affect its ability to operate in an efficient and effective manner. It is based on an analytical study of practices and performance of road agencies in seven countries (Botswana, Cameroun, Ethiopia, Ghana, Tanzania, Namibia and South Africa). The second approach is the British Standards Institution’s Publicly Available Specifi- cation (PAS55:2008) for the Optimized Management of Physical Assets as developed by the UK Institute of Asset Management, which applies to all types of infrastructure. It was recently used by the Association of Southern Africa National Road Agencies (ASANRA) in the Peer Review Benchmarking Study of Road Asset Management prac- tices in nine countries, i.e. South Africa, Botswana, Zambia, Lesotho, Malawi, Zimba- bwe, Namibia, Mozambique, and Tanzania. The PAS55 approach recognizes that the management of physical assets is inextricably linked to the management of other as- vii Road Management Policy– Road Agency Performance sets types, such as Human Assets, Information Assets, Intangible Assets and Financial Assets. However, PAS55 is more focused as these assets are only considered where they have a direct impact on the optimized management of the physical assets. The outcome of the two evaluations has provided valuable information to the road sector stakeholders in Africa in terms of the differing analytical approaches that may be used to quantitatively evaluate road agency performance in asset management. To provide an in-depth insight of the CRM and PAS55 approaches an analysis was under- taken to compare them based on results of road asset management capacity in four countries that participated in both studies, namely South Africa, Namibia, Botswana and Tanzania. The analysis has provided indication on how complementary elements of the two approaches could possibly be used to improve the framework of assessment of road agency capacity and performance based on key benchmarks and indicators for efficient road asset management. The analysis has also highlighted that such evalua- tions, whilst useful, need to be supplemented by the application of performance indica- tors to benchmark the agency’s operations, report objectively to management, and iden- tify areas for improvement. Performance management using these indicators should be viewed as an integral part of the overall road asset management process and is an area in need of further development. This working paper is intended to strengthen the knowledge base and awareness on road management good practices that is important for continuous advocacy on road sector reforms in African countries. It provides opportunity for road sector managers, decision-makers and professionals in Africa to broaden consultation for understand- ing the underlying principles of road asset management performance assessment. The information provided is fundamental for improving existing evaluation frameworks taking into account the contexts in which the road agencies operate. Notably, building on the strengths of both evaluation approaches, succinct recommendations have been given to incorporate the elements of PAS55, essentially the asset management aspects, in the more broadly based CRM evaluation approach in order to come up with a more comprehensive common framework for performance assessment within and across countries in Africa. ASANRA acknowledges the support provided by both the Africa Transport Policy Programme (SSATP) and the UK Department for International Development (DFID), through its Africa Community Access Programme (AFCAP), in undertaking the road agency performance evaluations. In recognition of the need to improve effi- ciency in road asset management in African countries ASANRA is keen to foster insti- tutionalization of good road management practices through continuous advocacy on viii Foreword road sector reforms in its member countries. This is important to rebuild the institu- tional memory on the basic principles of road sector reforms and commercialization of road management, particularly for the new generation of policy and decision mak- ers in the road sector. Also, ASANRA will encourage its member countries to deter- mine optimal strategies for improvement of road asset management, based on appro- priate performance monitoring frameworks with refined indicators for standard benchmarking. Such strategies will take into account institutional aspects that are pre- requisite for efficient and effective road institutions, and consider all elements im- portant for policy and decision-making processes. Equally, the African Development Bank has a long experience of road projects in sup- port of both development and maintenance. Good management is critical for the suc- cess of these projects, on one side for their sustainability and on the other side for their impact in terms of poverty and economic growth. The present document pro- vides a good foundation for governments and road agencies and departments to iden- tify areas of improvement. Merging the two approaches compared in the document provides the flexibility to adjust performance evaluation to the actual performance and capacity of road institutions, from a broad assessment based on the principles of commercialized road management to the more specific assessment of road asset man- agement. The African Development Bank is committed to continue this work initiated by SSATP and ASANRA to provide an evaluation tool available to all African coun- tries and beyond. This will be an essential complement to the efforts to bridge the re- maining road infrastructure gap on the continent. ix Acknowledgements This paper was prepared under the guidance of Camilla Lema, SSATP thematic leader for the SSATP Theme 2: sustainable institutional and financial arrangements for road infrastructure and rural and urban transport services. The paper was reviewed by Jean-Noel Guillossou and Justin Runji of SSATP, whose comments are gratefully acknowledged. The author is also grateful to Robert Geddes, former Technical Man- ager of the African Community Program (AFCAP) and Les Sampson, former facilita- tor of the RMI workshops in Sub-Saharan Africa and Asia for their contributions. Comments were also received during two presentations to the transport staff of the World Bank and the African Development Bank. The author is also grateful to Inje Oueslati who published the manuscript. xi Abbreviations and Acronyms AFCAP Africa Community Access Programme AGEPAR Association of African Directors of Roads AM Asset Management ARMFA African Road Maintenance Funds Association ASANRA Association of Southern African National Roads Agencies BB Building Block BSI British Standards Institute CEO Chief Executive Officer CRM Commercialized Road Management DFID UK Department for International Development FMS Financial Management System IAM Institute of Asset Management IDRC International Development Research Centre IOA Institutional Organizational Assessment MLP Model Legislative Provisions PAS Publicly Available Specification PI Performance Indicator RAG Red-Red/Amber-Amber/Green-Green RAM Road Asset Management RAMS Road Asset Management System RMF Road Management and Financing RMI Road Management Initiative SADC Southern Africa Development Community SATCC Southern Africa Transport and Communications Commission SSA Sub-Saharan Africa SSATP Africa Transport Policy Program xiii Executive Summary The potentially catalytic role of road transport in the socio-economic growth and de- velopment of many countries in Sub-Saharan Africa has not yet been fully realized despite the enormous investments that have been made in the road sector. In fact, backlogs in maintenance and inefficiencies in road sector operations in a number of Afri- can countries continue to impact adversely on many other sectors of their economies with coverage of routine and periodic maintenance averaging 65% and 54% respectively (SSATP RMI Matrix, 2012). The perceived poor results of road agencies have called into question their capacity for undertaking road asset management in an effective and efficient manner. This has led to various initiatives to evaluate such performance as a basis for determining those factors that aid or impede the agency’s achievement of their desired results. This would, in turn, provide a better understanding of what the agencies can or should change to improve their ability to perform in a more effective and efficient manner. Two organizational assessments of road agency performance have been undertaken in Africa in the past five years. The first assessment was based on the extent to which a road agency has been reformed in line with the Road Management Initiative (RMI) concept of commercialization of the roads sector (the Commercialized Road Man- agement (CRM) evaluation approach, the outcome of which is documented in SSATP Working Paper No. 92, 2012 – Progress on Commercialized Road Management in Sub- Saharan Africa). The second assessment was based on the extent to which a road agen- cy complies with the Publicly Available Specification (PAS) for the Optimized Man- agement of Physical Assets (the PAS evaluation approach as developed by the UK In- stitute of Asset Management and documented in PAS 55-1:2008). The outcome of this evaluation is documented in a report entitled Peer Review Benchmarking: Steps to The Future which was undertaken by the Association of Southern African National Roads Agencies (ASANRA) in 2014. The main objective of the present document on Road Management Policy: An Ap- proach to the Evaluation of Road Agency Performance is to analyze the CRM and PAS evaluation approaches for assessing road agency performance in road asset manage- ment. The document also considers how the two approaches could be amalgamated to enhance the evaluation of roads agency performance in terms of key benchmarks for xv Road Management Policy– Road Agency Performance good road asset management practice. Finally, the document reflects on indicators that could be incorporated in an appropriate asset management evaluation framework to: (1) measure the progress being made towards commercialization of road admin- istrations; and (2) monitor the performance of roads agencies in terms of the efficien- cy and effectiveness with which they undertake their responsibilities in the road sub- sector. The evaluation of the CRM and PAS approaches has been guided by the International Development Research Centre (IDRC)-Universalia framework for conducting an In- stitutional and Organizational Assessment (IOA) which has been successfully field tested in a number of developing countries (Lusthaus et al, 1999). In this approach, the performance of an organization is believed to hinge critically on the key dimen- sions pertaining to organizational capacity, organizational motivation and external environment, as illustrated schematically below. Institutional and organizational assessment (IOA) model The IOA model presented above implies that certain contextual forces drive perfor- mance, namely: the capacity of the organization, e.g. strategic leadership, human and financial resources, programming and process management and inter-institutional linkages); forces in its external environment, (e.g. the policy and regulatory environ- ment; and the internal motivation of the organization (e.g. organizational culture, history, mission, incentive systems). Largely, and maybe not surprisingly, these very forces are recognized to varying extents in both the CRM and PAS evaluation frame- works. xvi Executive Summary Key Findings and Conclusions Based on the above generic approach to organizational performance assessment, the following findings, conclusions and recommendations emerge from the evaluation of the CRM and PAS approaches. Comparison of CRM and PAS approaches for assessing road agency asset man- agement capacity i. The PAS and CRM frameworks for assessing road agency capacity in asset management have been developed in quite different ways, against quite dif- ferent environmental and cultural backgrounds and are quite differently fo- cused. ii. The CRM framework is the product of the RMI concept of commercializa- tion which was initiated in response to chronic shortcomings in the ability of road agencies in SSA to manage their road network in an efficient and ef- fective manner. The framework addresses key aspects of road asset man- agement that are perceived as being of critical importance for achieving pro- ficiency in this activity. The framework considers that reforms in the road sector, in line with the commercialization concept, are a necessary pre- requisite for achieving sustainable improvements in road asset manage- ment. Moreover, follow-up studies have shown that compliance with the requirements of the commercialization concept have generally resulted in improved conditions on the ground. iii. In view of its relatively broad focus, the CRM framework provides a com- prehensive basis for undertaking an overall evaluation of a road agency’s performance in asset management in that it addresses all those organiza- tional elements (the 4 Building Blocks pertaining to Responsibility, Owner- ship, Financing and Management) that affect the ability of roads agency to operate in an effective and efficient manner. iv. The PAS framework is an internationally recognized benchmark for road asset management that has been developed by the UK Institute of Asset Management and the British Standard Institute in response to demands from the industry in Europe for a standard for road asset management. This standard is based on compliance with an international specification for the optimized management of physical assets and the main focus is on man- agement of physical assets largely through the use of an asset management system including a supporting asset management strategy and related asset xvii Road Management Policy– Road Agency Performance management objectives and plans. Its application, so far, has been in rela- tively mature and developed road agencies in developed countries. v. In view of its relatively narrow focus, the PAS framework provides a neces- sary, but limited, basis for undertaking an overall evaluation of a road agen- cy’s performance in asset management. Indeed, many of the problems with road asset management stem from fundamental institutional and organiza- tional weaknesses, often inter-related. Consequently, although some im- provements result from addressing asset management issues, this limited approach is unlikely to improve overall performance unless other critical is- sues central to the concept of commercialized road management such as er- ratic funding and inadequate quality of staff are also addressed. Comparison of findings from application of the CRM and PAS frameworks i. The CRM framework has probed in some detail the factors associated with the achievement of the RMI commercialization concept as included in the four inter-related Building Blocks. Thus, it has pin-pointed fairly specifically the extent of an agency’s non-compliance with, for example, those elements of organizational structure, planning, management and financing of roads that critically affect the ability of the agency to fulfil its mandate in an effi- cient and effective manner. ii. As a result of the approach embedded in the CRM framework, it has been possible to not only indicate areas of weakness but, also, how to address them in terms of the required improvements/reforms in line with the four Building Blocks. iii. The PAS framework has probed quite extensively those elements of road agency operations related to the establishment of a road asset management system, including asset management strategy, asset management objectives and asset management plans. Coincidentally, one of the findings of the CRM study was that "Most road agencies are unable to operate their Road Asset Management Systems to produce reliable outputs in terms of optimal network strategies and programs”. South Africa and Namibia were the no- table exceptions to this finding. iv. As a result of the approach embedded in the PAS framework, it has been possible to pinpoint what needs to be done to achieve the PAS specification for road asset management, but not how to go about achieving this, which must be determined by the agency in accordance with its assessed needs. For example, the PAS identifies maintenance funding gaps as a major challenge xviii Executive Summary faced by road agencies but the root cause of the problem in most countries – insufficient, erratic, unsustainable funding –the manner of addressing the challenge, as detailed in the RMI commercialization concept, and the need to attain the elements of Providing Stable Financing, the third building block of the RMI concept, have not been identified as key issues. Assessments outcome based on application of CRM and PAS frameworks i. The CRM framework addresses a much more extensive set of parameters that are likely to affect road agency asset management capacity than is the case with the PAS framework. The CRM framework provides a very broadly based approach to determining road agency capacity in road asset manage- ment. The framework addresses not just the asset management system as- pects, but also other related aspects associated with the institutional ar- rangements, planning, management and financing aspects of roads, all of which critically affect road agency asset management capacity. In contrast, the PAS framework is more narrowly focused on the road asset manage- ment system aspects and much less so on other factors that affect road agen- cy asset management capacity. ii. From the strict point of view of asset management, both the CRM and PAS frameworks provide very similar results in terms of the overall rating of road agency capacity in road asset management. Scope for improvement of CRM framework The CRM and PAS frameworks offer complementarity in assessing road agency capacity in road asset management in that the more broadly based CRM frame- work can be refined by incorporating the more specifically focused asset manage- ment system aspects of the PAS framework. Performance Indicators i. Both the CRM and PAS evaluation outcomes are quantitatively based and can be used as performance indicators that measure progress in commer- cialized road management. ii. The RMF and SSATP indicators proposed in Working Paper No. 95: Transport Governance Indicators for Sub-Saharan Africa, together with the CRM and PAS indicators, offer valuable inputs to the development of a measurement framework and related indicators for measuring and report- ing progress in the efficiency and effectiveness of road agency performance. xix Road Management Policy– Road Agency Performance However, they need to be supplemented with other indicators to cover oth- er aspects of road agency performance including institutional effectiveness, sectoral effectiveness and provision efficiency. Moreover, the indicators need to be defined and specified in a unique manner and the supporting da- ta for calculating them need to be harmonized to ensure consistency in their use. Key Recommendations The following key recommendations emerge from this review. i. The CRM evaluation framework should be improved (primarily in its aspect of promoting commercial management) by incorporating those elements of the PAS evaluation framework that are not currently captured so as to pro- duce an enhanced framework for evaluating road agency performance in as- set management as illustrated below. Such a framework would incorporate the strengths of the individual frameworks, including all four Building Blocks embedded in the RMI commercialization concept. ii. The various elements of both the CRM and PAS evaluation frameworks should first be reviewed by key stakeholders as a basis for improving the CRM framework. This process will enable stakeholders to reach consensus on the structure of the new framework. By so doing, it will engender their full buy-in and ownership of the final product and, as a result, is likely to lead to more widespread use in future. xx Executive Summary Improvement of CRM evaluation framework using complementary PAS elements iii. The RMF and relevant SSATP indicators proposed in Working Paper No. 95: Transport Governance Indicators for Sub-Saharan Africa, should be developed into a performance measurement framework that measures not only progress with commercialization, via the four Building Blocks, but also looks beyond at other performance indicators, as listed below, each with their own perfor- mance indicators: a. Performance Indicator 1. Measures Progress with commercialization b. Performance Indicator 2. Measures Institutional effectiveness c. Performance Indicator 3. Measures Sectoral effectiveness d. Performance Indicator 4. Measures Provision efficiency e. Performance Indicator 5. Measures Infrastructure provision iv. In order to ensure successful integration of the PIs into road agency man- agement processes, a number of measures should be observed, including in- volving all stakeholders in their development, identifying the need for addi- tional data collection, data management, and analytic tools to support the se- lected indicators, estimating the cost of operating and maintaining the meas- urement framework and securing a matching budget. xxi 1. Introduction Context Road transport is the most pervasive of all transport modes in Africa and is central to the socio-economic growth and development of all countries in the region. To be ef- fective as a catalyst for development, road provision must be institutionally, economi- cally, financially, socially and environmentally sustainable. This remains a challenge in most African countries despite the enormous investments that have been made in the provision of road infrastructure over the past few decades. Indeed, backlogs in maintenance and inefficiencies in road sector operations in a number of countries continue to impact adversely on many other sectors of national economies with cov- erage of routine and periodic maintenance needs averaging 65% and 54% respectively (SSATP RMI Matrix, 2012). The ways for the road sector to achieve its potentially catalytic role in national eco- nomic development are now well known as a result of the Road Maintenance Initiative (RMI) undertaken as long ago as 1987 by the Africa Transport Policy program (SSATP). The RMI initially perceived lack of maintenance as the overwhelmingly im- portant constraint to the development and sustainability of road systems in SSA and, within that aspect, stability of financing was perceived as the central problem. In 1997, the name of the RMI was modified to Road Management Initiative – a change that represented the realization that stable financing in itself is not sufficient for en- suring proper maintenance of the road network, and that aspects of institutional structure, planning, management, and the external environment all strongly influence a road agency’s ability to manage the road asset in an effective and efficient manner. This approach was in full recognition of the main factors that adversely affected the efficiency of road agencies in the 1990s, as more recently re-stated by Queiroz and Kerali (2010):  outdated management structures  lack of clear responsibilities  human resource constraints  weak management information systems 1 Road Management Policy– Road Agency Performance  inadequate financing  perception of roads as a public good In view of the above, the RMI focused on finding ways and means of reversing the process that was trapping governments in the cycle of road building, roads deteriorat- ing prematurely and roads being rebuilt long before the end of their design life. The key concept to emerge from the RMI program was that of commercialization of the road sector (Heggie, 1995; Heggie and Vickers, 1998), i.e. using market concepts and discipline in road management and introducing a fee-for-service element in the fi- nancing. However, since roads are a public monopoly, and ownership of most roads will remain in government hands for the foreseeable future, commercialization re- quires complementary reforms in four key areas – the so-called four Building Blocks (BB) (Figure 1). Figure 1 – Four Building Blocks necessary for commercialized road management Essentially, the RMI defines a number of regulatory essentials and provides guidelines, which have to be accommodated to ensure effective and efficient road management and funding irrespective of what the institutional framework will ultimately look like. These regulatory essentials may be summarized as follows:  a clear demarcation and allocation of authority and responsibility for road financing and management;  significant private sector participation, and in particular from transport users, in road financing and management decision-making forums;  adoption of appropriate financing principles and practices in order to secure: - adequate and stable sources of funding through road user charges; - arrangements to allocate funds in an unbiased and transparent manner to relevant road authorities/agencies; 2 Introduction - performance-based fund allocation; and - adoption of sound commercial management principles and practices. The RMI vision was developed on the basis of extensive discussions between stake- holders and sector specialists. It recognizes from historical experience that many of the problems with road management are caused by institutional and organizational weaknesses, which are often inter-related. Thus, although some improvements are likely from addressing a particular aspect of a road agency’s operations, much greater benefits are likely to occur by ensuring that each change fits within a consistent and broader structure of improvements, as represented by the four BBs. The RMI vision has been widely accepted as a basis for policy reform in many coun- tries. In Africa, it is embedded, for example, in the Southern African Development Community (SADC) Protocol on Transport, Communications and Meteorology (SADC, 1999) as well as the SADC Model Legislative Provisions on Road Network Financing and Management (SATCC, 1999). Over the past two decades, a number of countries have embarked on wide-ranging road sector reforms largely in accordance with the RMI vision. This has included the establishment of independent road funds and, to a somewhat lesser extent, autonomous or semi-autonomous road authorities. Table 1 shows the status of road sector reforms that have taken place to date in Africa. Table 1 – Countries with established Road Funds and Road Authorities in Africa Source: RMI Matrix (2006) Country with both a Road Fund and a Road Authority Subsequently to the long involvement of the SSATP in promoting the RMI principles, three regional road associations were mandated to advocate the RMI principles and monitor their application. These associations are: i. The Association of Southern Africa National Roads Agencies (ASANRA) was established in 2001 and is comprised of representatives of national road agencies, educational institutions, industry/private sector and other interest- 3 Road Management Policy– Road Agency Performance ed parties. The organization’s main goal is enhancing regional policy coordi- nation and road transport systems integration in order to improve intra- regional road transport efficiency and lower transport costs. ii. The African Road Maintenance Funds Association (ARMFA) was established in 2003 and is comprised of the heads of national Road Funds in West, Cen- tral, Southern and Eastern Africa. The organization’s main goals include de- veloping an information sharing network on best road maintenance financ- ing practices in Africa and on the functioning of Road Funds; supporting the promotion and strengthening of links between African Roads Funds; and, ensuring the medium term sustainability and harmonious development of Road Maintenance Funds. iii. The Association of African Road Managers and Partners (AGEPAR) estab- lished in 1994 is comprised of the heads of road departments and agencies in West Africa. The organization’s main goals include: promotion of the inte- gration of road networks, harmonization of standards for construction and maintenance and performance improvement in the road sector. From the Africa Infrastructure Country Diagnostic study undertaken in Sub-Saharan Africa, (Gwilliam et al, 2008), it was found that the RMI suite of reforms had had a discernible impact on outcomes. Countries with well-financed road funds were found to do significantly better at capturing resources for maintenance. Moreover, countries with road funds and road agencies do significantly better at converting resources into road quality. The study concluded that the advance of institutional reforms, though impressive, was incomplete in many cases – a finding that is corroborated by the SSATP as reported in its Working Paper No. 92, which concluded that progress in the institutional reforms may best be described as “a road partially travelled”. Moreover, it has become increasingly apparent that lack of continued advocacy of the principles of the RMI in the Africa region has resulted in loss of institutional memory as a result of which the current generation of road sector stakeholders is largely unaware of the rationale behind the initiative. In view of the far-reaching and often costly reforms that have taken place in the road sector of many African countries during the past two decades, it is important that an evaluation of the organizational performance of road agencies be carried out to better understand the impact of the reforms, learn lessons from these experiences and identi- fy how agencies can or should change to improve their ability to perform. To this end, two major studies, based on somewhat different approaches to assessing road agency asset management capacity, were undertaken. 4 Introduction i. Progress on Commercialized Road Management in Sub-Saharan Africa. (2012). M. Pinard, SSATP Working Paper 92. ii. Peer Review of Road Asset Management Practices in the Southern Africa De- velopment Community (SADC) Region. (2014). I.E.S Asset management, BV, Netherlands. Objectives With both studies having been carried out in the same geographic area in Africa and having used two different methodologies, ASANRA and the SSATP took the oppor- tunity to learn from the dual exercise with the objective of providing guidance for the evaluation of the performance of road agencies. Against this background, the main objective of this working paper is as follows: i. To compare and contrast the approaches adopted in the above studies to as- sess roads agency performance in road asset management including aspects of complementarity and divergence. ii. To identify how the approaches could possibly be amalgamated to enhance the evaluation of road agency capacity and related performance in terms of key benchmarks for good road asset management practice. iii. To propose indicators that could be incorporated in an appropriate asset management evaluation framework to: (1) measure the progress being made towards commercialization of road management, and (2) monitor the per- formance of roads agencies in terms of the efficiency and effectiveness with which they undertake their responsibilities in the roads sub-sector. Structure The publication is structured as follows: Section 1 (this section): Presents the background to the working paper, including its purpose and structure. Section 2: Considers the need for a holistic approach to assessing road agency perfor- mance in road asset management. Section 3: Provides an overview of the alternative approaches to assessing roads agen- cy capacity in asset management and their ability to pinpoint those key elements that contribute to effective and efficient road asset management. 5 Road Management Policy– Road Agency Performance Section 4: Compares and contrasts the outcomes of the assessment of road agency capacity in road asset management produced by the alternative approaches and uses data from three countries to compare the outcomes produced by the two methods. Section 5: Considers various performance measures and indicators that could be adopted to monitor and report upon on a continuing basis road agency performance in asset management. Section 6: Presents the findings, conclusions and recommendations of the study in- cluding a plan of action to implement the best aspects of both approaches to assessing roads agency capacity in road asset management. 6 2. A Holistic Approach to Assessing Road Agency Performance Challenges faced by Road Agencies in Africa African governments are placing increasing pressure on road agencies to improve the efficiency of, and accountability for, the management of their road networks. In most African countries, the road network constitutes one of the largest public assets and is generally wholly government-owned. Typically, roads agencies are entrusted with maintaining, operating, improving, replacing and preserving the road asset while, at the same time, carefully managing the scarce financial and human resources needed to achieve these objectives. This must be accomplished under the close scrutiny of the public who pay for, either via general taxes or road tariffs, and are regular users of the road network, and who increasingly demand improved levels of service in terms of safety, reliability, environmental impact and comfort. Road agencies face significant external, internal institutional and technical chal- lenges that affect their ability to operate in a commercialized manner. In general, these challenges relate to the following: i. Governance issues ii. Weak legislative framework upon which RMI reforms can be based iii. Fragmented approaches to planning of road sector programs iv. Poorly developed commercial sector v. Lack of vibrant and competitive domestic consultant and construction industries, which limits the extent to which non-core activities can be contracted out to the private sector vi. Problems with retrenchment of large numbers of in-house staff Of the above challenges, governance is arguably the most important one. Good governance, or the lack of it, can have a profound effect on the ability of road agencies to operate in a commercialized manner. Good governance in the transport sector is now widely recognized as being crucial for attaining sustainable 7 Road Management Policy– Road Agency Performance improvements in road asset management. In a recent study commissioned by SSATP (Christie, Smith and Conroy, 2012 in SSATP Working Paper No. 95), the following transport governance issues were identified across four pilot countries: i. Political interference in projects and in key governing board appoint- ments ii. Limited or non-existent integrated transport sector policy iii. Identification of new and strengthened institutional and regulatory ar- rangements, together with continued evidence of on-going blurred boundaries, unrealistic budgets, and no objective criteria for prioritiza- tion iv. Long-term under investment and prioritization in maintenance The above issues all reflect a poor governance environment, which is not condu- cive to improving the efficiency and effectiveness of road agency operations. These issues were identified in the process of developing the RMI vision more than two decades ago! They indicate clearly that poor performance in the road sector is due to far more than mere proficiency in asset management practice and, as a result, any evaluation framework should address all factors that affect the ability of a road agency to operate in an efficient and effective manner. Factors Affecting Road Agency Performance Capacity is a critical factor for a road agency to perform efficiently and effectively. Connolly and Lukas, 2002, identified the following components of capacity that are deemed to be critical for high performance in any agency: i. Mission, Vision and Strategy: Need for vital mission and clear understand- ing of its identity ii. Governance and Leadership: Need for a board of directors, with defined governance practices to effectively oversee the organizations’ policies, programs and operations iii. Finance: Need for the organization to successfully secure support from a variety of sources to ensure that its revenues are diversified, stable, and sufficient for achieving its mission and goals iv. Internal operations and management: Need for efficient and effective op- erations and strong management support systems with financial opera- tions responsibly managed and reflecting sound accounting principles 8 Approach to Assessing Road Agency Performance v. Program Delivery and Impact: Need to operate programs that demonstrate tangible outcomes commensurate with the resources invested vi. Strategic relationships: Need to engender respect and maintain strong connections with its constituents including participation in strategic alli- ances and partnerships that significantly advance their goals and expand their influence Several international studies of road transport infrastructure and financing (Heg- gie and Vickers, 1998; Queiroz and Kerali, 2008; Pinard, 2012) have identified fac- tors that impact on efficiency and effectiveness of road agencies, all of which are in accordance with the RMI commercialization concept: i. Establishing appropriate institutional structures and relationships with stakeholders ii. Separating the client and supplier functions iii. Separating the client and supplier organizations iv. Privatizing the supplier organizations v. Establishing an executive agency or a commercialized (client) organiza- tion vi. Ensuring user participation through oversight boards vii. Improving management information systems viii. Seeking additional, sustainable sources of financing Figure 2– Hierarchy of management issues: The Brooks Pyramid Source: Robinson, Danielson and Snaith, 1998 Attainment of the above requirements has been shown from other studies (e.g. Brooks, Robinson and O’Sullivan, 1989) to be insufficient to ensure satisfactory performance in a road agency. Instead, Robinson, Danielson and Snaith, 1998, argue that successful performance depends on three fundamental, hierarchically 9 Road Management Policy– Road Agency Performance interrelated factors that take the form of a pyramid as in Figure 2. What the pyra- mid indicates is that institutional development should be undertaken from the bottom up. Thus, for technical improvements to be successful, sufficient institu- tional capability must exist. Furthermore, unless the external context (e.g. govern- ance) is supportive, sustainable improvements in the organization are unlikely to be achieved (Stankevich and Guillossou, 2008). Road Asset Management Within their broad mandate, road agencies face a complex, multi-faceted array of operational issues related to road asset management. At each turn, crucial deci- sions must be made, and each involves expenditure of valuable resources. For example: i. What is the level of maintenance and rehabilitation funding required to maintain the road network in an optimal condition and what is the corre- sponding level of road user charges that should be levied? ii. What will happen to road surface conditions and structural capacity if an optimum budget is not available and what will be the resulting excess us- er costs? iii. What choices of maintenance and rehabilitation actions and budgets will most effectively bring pavement conditions to an optimal level? iv. In short, what should be the highway budget for various activities, where should the money be spent, and what exactly should be the road im- provement action? The technical issues posed above are difficult but important ones. They are diffi- cult not only technically, but also because of the sophistication of the road transport system, the extent of the road network, the complexity of relationships, and the political realities which are always present in road system management and decision-making, and which all pose a major challenge for any roads agency manager to grapple with. Because of this complexity, road agency managers are placing increased reliance on the use of an appropriate “systems engineering” approach in combination with modern day “management” techniques which, in practice, combine to produce a tool which has become almost, if not, indispensable to today’s generation of road managers – a Road Asset Management System. 10 Approach to Assessing Road Agency Performance The term “asset management” is used differently by different practitioners. For the purpose of this paper, the definition given by the British Standards Institution for their Publicly Available Specification (PAS) (IAM, 2008) which applies to all types of infrastructure is as follows: “systematic and coordinated activities and practices by which an organization optimally and sustainably manages its assets and asset systems, their associated performance, risks and expenditures over their life cycles for the pur- pose of achieving its organizational strategic plan”. This definition recognizes that asset management is related to delivering business goals through a combination of management, financial, economic, engineering and other related activities with the objective of providing the required level of service in the most cost-effective man- ner Therefore asset management can be viewed conceptually as the central overlap of the key activities shown in Figure 3. Figure 3: Conceptual components of Asset Management Source: Robinson 2008. 11 Assessment of Road Agency Performance in Asset Management 3. Assessment of Road Agency Performance in Asset Management Introduction As outlined in Section 2, the performance of a road agency is dependent on a wide range of factors that affect the efficiency and effectiveness of its operations. This suggests that the assessment of road agency performance should be undertaken in a holistic manner and should evaluate all the factors that are likely to affect such capacity and related performance. In this regard, the Institutional and Organiza- tional Assessment Model (IOA Model) developed by the Institutional Develop- ment Research Centre - IDRC), provides a well-tested framework for organiza- tional performance assessment that has been successfully applied by the IDRC in a number of developing countries (Lusthaus, et al, 1999). In the IOA evaluation framework, the performance of an organization is viewed as a multi-dimensional balance between its effectiveness, relevance, efficiency, and financial viability. The framework also presumes that the organizational perfor- mance should be examined in relation to the organization’s capacity, motivation, and external environment. In the schematic representation of the IOA framework in Figure 4, performance is defined in terms of effectiveness (mission fulfilment), efficiency, ongoing relevance (the extent to which the organization adapts to changing conditions in its environment), and financial viability. The framework implies that certain contextual forces drive performance, namely: the capacity of the organization, forces in its external environment, and the internal motivation of the organization. Figure 4: Institutional and Organizational Assessment (IOA) Model 13 Road Management Policy– Road Agency Performance Source: IUCN 1999. Against the above generic approach for evaluating the performance of an organiza- tion, the next section considers two frameworks, based on somewhat different approaches, used recently to evaluate the asset management capacity and related performance of a number of roads agencies in the SADC region. These frame- works are: i. The CRM framework ii. The PAS framework The CRM Framework for Assessing Road Agency Performance in Asset Management The CRM framework is based on the extent to which the agency has been re- formed in line with the RMI concept as discussed in Section 1. The underlying premise is that such reform is a necessary pre-requisite for establishing institution- al arrangements that are conducive to effective and efficient service delivery as outlined in the Sourcebook on Institutional Development for Utilities and infra- structure (Heggie and Kerali, 2000). The evaluation framework is based on a checklist for examining road agency performance and, in so doing, includes the basic elements of the four Building Blocks (BB) that cover all aspects of road ser- vice delivery normally expected of a road agency, each with a specific aim related to good practice. Based on the broad framework for assessing road agency asset management capac- ity presented above, a survey questionnaire was developed to cover all the elements of the four BB associated with commercialized road agency operations. Each ques- tion asks whether a particular feature of road management in the evaluated agency corresponds to a good practice as promoted by the RMI through the four BB. Most questions offer several potential answers, which move progressively from “Unsatisfactory” through to “Excellent” in terms of prevailing practice. When a particular feature of the road sector is checked against the checklist, the answer provides two sets of information: i. How does the road sector in this country score on this feature, measured in terms of Rating 1 (unsatisfactory practice) to Rating 5 (Excellent practice) ii. If less than Rating 5, what progressive reforms are required to take this fea- ture from where it is towards a higher-rated good practice. 14 Assessment of Road Agency Performance in Asset Management Table 2. CRM framework for assessing road agency performance in asset management Building BB.1 Responsibility Block Element 1 Institutional Framework Aim Separate management and financing of roads. Establish by Acts of Parliament a functionally autonomous Road Agency and Road Fund Administration as separate juristic Good practice persons responsible respectively for road management and financing, overseen by separate Boards with public/private sector participation in which the private sector is in the majority. Element 2 Organization Create a coherent organizational structure with functional separation of roles and clearly assigned responsibilities for Aim managing different parts of the road network. Separate client and supplier roles; formal assignment of responsibilities amongst roads agencies; establishment of a func- Good practice tionally classified road inventory. Building BB.2 Ownership Block Element 3 Oversight management Aim Build constituencies with a strong vested interest in sound road management. Good practice Involve road users in road management and decision-making to help win public support for secure and stable funding. Building BB.3 Financing Block Element 4 Secure and stable financing Aim Secure an independent, sustainable source of road funding through incremental expansion of road user charging system. Good practice Establish by an Act of Parliament a dedicated Road Fund that serves as the depository of the road user charge Element 5 User representation on board Aim Engender strong stakeholder support for putting roads on a fee-for-service basis. Good practice Establish a broad-based private/public board with majority private sector. Element 6 Road user charges Aim Recognition of the “user pays” principle by putting roads on a fee-for-service basis to generate additional revenue separate from the vagaries of the government’s budget Good practice Levy an appropriately road user charge to recover from road users the economic cost of road usage and to be used for eventually covering all costs of maintaining main roads and part of the costs of maintaining urban and rural roads. Element 7 Revenue collection Aim Direct transfer of road user charge to the dedicated Road Fund account. Good practice Ring fence road user charges which are deposited directly into the Road Fund. Element 8 Revenue allocation Aim Transparent allocation of funds to various road agencies. Good practice Develop sound criteria for allocating funds to various tiers of the road network. Element 9 Accounting for Road Fund Aim Ensure that all funds are accounted for in a sound and transparent manner. Good practice Establishment of a cost accounting system. Element 10 Independent auditing Aim Ensure that all revenues due to the Road Fund are collected and reported; that no funds are diverted for non-authorized purposes; and safe custody of funds. Good practice Undertake regular, independent, financial auditing of the Road Fund and make report public. Building BB.4 Management Block Element 11 Degree of autonomy Aim Strengthen managerial accountability. Good practice Appointment of the CEO by the Board with an annual Contract Plan including clear objectives and performance targets. Element 12 Management structure and operating procedures Aim Strengthen the management of roads by the adoption of sound business practices. Good practice Ensure that the CEO and line managers generally operate along commercial lines. Element 13 Human resources Aim Recruit and retain capable staff by offering competitive salaries. Good practice Adopt terms and conditions of employment for the CEO and staff comparable to those in private sector organizations. Element 14 Financial management Aim Develop the ability to present a clear picture of the agency’s overall financial health. Good practice Maintain commercial accounts by the national roads agency and the operation of a cost accounting system that provides at least an income and asset statement. 15 Road Management Policy– Road Agency Performance Element 15 Management information systems Aim Maintain objectivity in setting priorities & evaluating appropriate technology for road works. Good practice Operate an appropriate management system for planning and setting priorities for investment and maintenance. Element 16 Procurement and tender procedures Aim Contract out implementation of road works to the private sector in an efficient manner. Good practice Contract out most of the design, supervision and implementation of road works and the existence of well-developed procurement procedures for doing so. Element 17 Quality control Aim Adhere to auditing procedures to ensure that the public gets value for money from road spending. Good practice Check regularly the quality of construction and maintenance works through independently undertaken technical audits. Element 18 Technical auditing Aim Measure effectiveness of road agency operations to check whether it fulfils its role as effectively as possible. Good practice Undertake independent auditing of road works and make report public. An example of an evaluation questionnaire pertaining to BB.2: Ownership is presented in Table 3. Once filled, the questionnaire provides the basis for holding detailed consul- tations with stakeholders in a particular country on that topic to identify actions needed to improve the performance of the country in road management. Table 3. Example of CRM evaluation questionnaire BB.2. Ownership Unsatisfactory Satisfactory Key Objective: Empower & encourage the public to play an active role in the man- Excellent Good Poor agement of roads and win their support for road contributing to a Road Fund. Element: Oversight management Issue: Supervision of the Roads Authority Questions: Answer No Yes Rating score 1 2 3 4 5 Q.1 (a) Does the RA have any kind of oversight Board? If no, go to next section.  (b) Does the Board include members from road users and the business community?  (c) Are the private sector members of the Board in the majority?  (d) Are members nominated by the organizations they represent?  Q.2 (a) Does the Board have Terms of Reference?  (b) Are the Terms of Reference published in the Gazette or similar?  (c) Are the terms of Reference written into regulations under the Roads Act?  (d) Do the Terms of Reference include a Code of Conduct?  Q.3 (a) Is the Board chairman appointed by the Minister without consultation? or (b) Is the Board chairman appointed by the Minister after consulting the Board? or  (c) Is the Board chairman elected by the Board? Q.4 (a) Does the Board meet on an annual basis? or (b) Does the Board meet at least every six months or (c) Does the Board meet at least every three months? or  (d) Does the Board meet at least every per month? Q.5 (a) Are the workings of the Board ever open to the public (shareholders)?  (b) Are the minutes of the Board meetings ever made available to the public?  (c) Are selected embers of the public ever entitled to attend Board meetings?  Total score 4 0 9 20 5 Tick only shaded cells as applicable Total score = T = 38 Max possible score = 54 Rating (%) = T/54 * 100 = 38/54*100 = 70.4 16 Assessment of Road Agency Performance in Asset Management Measurement of road agency capacity in asset management Following completion and analysis of the outcome of the CRM evaluation ques- tionnaire, the capacity of a road agency to undertake efficient and effective asset management is determined by the extent to which it complies with the require- ments of the BB. This approach assess road agency capacity and related perfor- mance through a set of performance indicators (see Section 5) in which the rating is expressed on a chosen graduated scale ranging from Unsatisfactory (rating 0-20) to Excellent (rating 80-100) as illustrated in Figure 5 1. Figure 5 – Example of CRM measurement of road agency performance By way of example, the following broad conclusions may be drawn from Figure 5 regarding progress on commercialization of a road agency’s operations in terms of the extent of implementation of the reforms embodied in the three BB. (Note: The approach is equally applicable to evaluating all four BB). Table 4. Example of reporting on general progress on commercialization Progress on implementation Rating (%) Evaluation of Building Block BB.1 70 Good BB.2 40 Poor BB.4 64.3 Good BB.1 + BB.2 + BB.4 63 Good 1 Only three of the BB were considered in the CRM study (BB1, BB2 and BB4), a study focusing on BB3 having already been carried out by SSATP (Benmaamar, 2006). 17 Road Management Policy– Road Agency Performance Table 5, which may be developed from the information contained in Table 4, pro- vides a detailed assessment of the progress made on each element of each Building Block. In so doing, those features that require particular attention can be isolated and addressed, as appropriate. Table 5. Example of reporting on detailed progress on commercialization Progress on Commercialisa- Building Block Feature tion Unsatisfactory BB.4 Quality Control Poor BB.2 Ownership Satisfactory BB.1 Institutional management Degree of autonomy Management structure and procedures Procurement procedures Good BB.1 Institutional framework Human resources Financial management Excellent BB.4 Management Information Systems The PAS Framework for Assessing Road Agency Performance The Publicly Available Specification (PAS) was developed by the UK Institute of Asset Management (IAM) and the British Standards Institution in 2004 in re- sponse to demands from the industry for a standard for asset management (Insti- tute for Asset Management, 2008). In order to ensure consistency with other relat- ed management system standards, it was considered that asset management would best be standardized as a specification, with the information on implementing asset management distilled into key requirements. The criterion for including such requirements has been that, without them, the asset management system would be regarded as deficient. The Specification is applicable to any organization where physical assets are a critical factor in achieving its business. The PAS is primarily designed to support the delivery of an organizational strate- gic plan in order to meet the expectations of a variety of stakeholders. Thus, the plan becomes the starting point for developing the organization’s asset manage- ment policy, strategy, objectives and plans. These, in turn, direct the optimal com- bination of life cycle activities to be applied across the diverse portfolio of assets. Figure 6 presents an overview of the asset management system and its relationship to the organizational strategic plan and stakeholder expectations. 18 Assessment of Road Agency Performance in Asset Management Figure 6. PAS framework for asset management Source: IAM 2008 The PAS framework is essentially a measurement/compliance tool, which, as indi- cated in Figure 6, focuses on the management of physical assets and asset systems in accordance with an over-arching asset management policy. The framework measures the extent to which a road agency complies with the PAS Specification for the Optimized Management of Physical Assets. This specification provides key per- formance indicators across 28 aspects of good asset management (Figure 7). It is important to note that the requirements of PAS are prescriptive only to the extent that they define what has to be done, not how to do it. The method of achieving each requirement is for the requirements of PAS are prescriptive only to the extent that they define what has to be done, not how to do it. The method of achieving each requirement is for the organization to determine in accordance with its as- sessed needs. 19 Road Management Policy– Road Agency Performance Figure 7. PAS management system structure Source: IAM 2008 PAS 55 recognizes that the management of physical assets is inextricably linked to the management of other asset types, such as Human Assets, Information Assets, Intangible Assets and Financial Assets. However, these assets are only considered where they have a direct impact on the optimized management of the physical assets. As is apparent from Figure 7, the main requirements within PAS call for the organization to establish an asset management system, including asset manage- ment strategy, asset management objectives and asset management plans. Other organizational requirements that could affect performance, e.g. institutional mandates and responsibilities, financial sustainability, human resources, or the external environment, e.g. stakeholders such as road users or service providers, are not addressed. 20 Assessment of Road Agency Performance in Asset Management In order to assess the “level of maturity” or capacity of the road agency in relation to an aspect of road asset management, PAS 55 developed a questionnaire which covers all 28 aspects of the PAS requirements indicated in Figure 7. A typical ex- ample of the questions pertaining to asset management strategy (Item 4.3.1 in Fig- ure 7) is presented below in terms of the requirements to be met by the organiza- tion for various maturity levels. Table 6. Example of PAS evaluation questionnaire Legend: 0 = Unsatisfactory; 0.1- 1.0 = Poor; 1.1- 2.0 = Satisfactory; 2.1- 3.0 = Good; 3.1- 4.0 = Excellent 21 Road Management Policy– Road Agency Performance Following completion and analysis of the outcome of the PAS evaluation ques- tionnaire, a road agency’s asset management capacity can then be measured in a quantitative manner as illustrated in Figure 8. Figure 8. Example of PAS measurement of road agency performance in asset management By way of example, the following broad conclusions may be drawn from Figure 8 regarding the level of Asset Management (AM) maturity of the roads agency. Table 7 – Assessment of PAS asset management maturity Clause No. 2008 Clause AM Rating AM Evaluation 4.1 General requirements 1.0 Poor 4.2 AM policy 0 Unsatisfactory 4.3 AM strategy, objectives & plans 0.9 Poor 4.4 AM enablers and controls 1.2 Satisfactory 4.5 Implementation of AM plans 2.7 Good 4.6 Performance assessment & 1.4 Satisfactory improvement 4.7 Management review 1.2 Satisfactory Overall, in this example, the shape of the radar graph indicates a relatively imma- ture organization as regards to asset management as defined by PAS. 22 Assessment of Road Agency Performance in Asset Management Comparison of CRM and PAS Approaches for Assessing Road Agency Perfor- mance in Asset Management Similarities and differences in approach of the CRM and PAS frameworks for as- sessing road agency performance in asset management are summarized in Table 8. Table 8–Comparison of CRM and PAS approaches for assessing road agency performance Similarities  Use of objective, measurable criteria in various aspects of road asset management.  Use of a far-ranging questionnaire to ascertain the extent of compliance of the agency with a bench- mark, the asset management-related standard.  Can evaluate, and quantitatively rate, the capacity of the agency to undertake road asset management in accordance with the benchmark criteria considered.  Are not linked to any road indicators that can be used to assess the level of road agency capacity, mon- itor results and act on the findings. Differences CRM framework PAS framework  Developed on the basis of extensive and wide-  Developed by an international organization ranging consultations with stakeholders in the (IAM) for international stakeholders SADC region.  Applies specifically to environments in which  Applies implicitly to environments in which the the road agency is perceived to be in need of organization is not perceived as being in need of structural reform in line with the RMI commer- structural reform as a necessary pre-requisite for cialization concept including attainment of the achieving proficiency in road asset manage- requirements of the related 4 BB. ment.  Applies specifically to the management of phys- ical assets and only considers other asset types, such as human, financial and information assets, where they have a direct impact on optimized management of the physical assets.  Specifically addresses a large number of road  Applies to any organization dealing with asset sector-specific factors (external, internal insti- management, not specifically in the road sector, tutional and technical) that are perceived in and thus does not specifically address a number the Africa region as directly influencing road of roads sector-specific attributes that, from his- agency performance in road asset manage- torical experience, are perceived in the SADC re- ment. gion as being desirable, if not necessary, for achieving lasting improvements in road asset management proficiency.  Addresses a few key issues that are not addressed in the CRM framework, and are now perceived as a necessary aspect of proficient road asset manage- ment, including risk management processes, meth- odology and assessment; contingency planning.  Produces outputs that pin point gaps in asset  Produces outputs that define what has to be management capacity and indicates how to done, but not how to do it. The method of close these gaps in terms of the agency’s at- achieving each requirement is for the organiza- tainment of the requirements of the four BB. tion to determine in accordance with its as- These outputs can be aggregated to allow an sessed needs. These outputs can be aggregated overall rating of the agency to be obtained. to allow an overall agency rating to be obtained. 23 4. Outcomes of Alternative Approaches for Assessing Road Agency Performance in Asset Management Introduction This chapter compares the alternative approaches that were both used for assessing road agency capacity in asset management, as described in Section 3, in a number of countries in the southern African region as follows: i. The CRM framework: The assessment was undertaken by a consultant who discussed and rated each of the questions contained in each of the four BB, as illustrated in Table 2, based on detailed discussions with a cross-section of senior staff of the road agencies. The countries surveyed are shown below whilst the outcome of the assessment is contained in SSATP Working paper No. 92 (Pinard, 2012).  Botswana  Namibia  Cameroon  South Africa  Ethiopia  Tanzania  Ghana ii. The PAS framework: The assessment was undertaken initially on a self- assessment basis by road agency staff after undergoing training in the PAS 55 assessment methodology. The outcome of the assessment was then veri- fied by an independent review panel and adjusted, where appropriate, through joint panel interviews in each country. The countries surveyed are shown below whilst the outcome of the assessment is presented in a report entitled: Peer Review Benchmarking: Steps to the Future. I.E.S Management BV (January, 2014).  Botswana  Mozambique  Tanzania  Lesotho  Namibia  Zambia  Malawi  South Africa  Zimbabwe 25 Road Management Policy – Road Agency Performance Outcomes from Application of CRM and PAS Frameworks The outcomes of the application of the CRM and PAS frameworks used to evalu- ate road agency capacity in road asset management are compared at two levels:  First at the level of the general findings of each evaluation of all the coun- tries surveyed.  Second at the level of the detailed findings of the evaluation for three of the countries surveyed (South Africa, Tanzania and Botswana). General findings and recommendations emanating from application of the CRM framework The general findings derived from the application of the CRM framework are summarized as follows:  Progress on commercialization of road management practices varies con- siderably from just satisfactory to excellent.  There is generally lack of genuine autonomy of Roads Authorities from parent ministries.  Roads Boards are still predominantly public sector dominated and road user interests are not adequately articulated.  Good governance mechanisms in many Roads Boards operations are generally inadequate.  Board members are often not abreast of the fundamentals of commercial- ized road management and financing.  Many road agencies fulfil aspects of the “supplier” function and under- take varying amounts of non-core activities.  Lack of Procedures Agreements between some Road Authorities and Road Funds adversely affect the implementation of Road Network Man- agement Plans.  Performance measurement is not carried out in most road agencies.  There is significant over-staffing in traditional Government roads institu- tions in terms of the number of staff responsible for managing 100 km of roads compared to more commercially oriented agencies. 26 Alternative Approaches for Assessing Road Agency Performance  Salaries are still not market based in most road agencies, which often find it difficult to attract and retain competent staff.  Most road agencies are unable to operate their Road Asset Management Systems to produce reliable outputs in terms of optimal network strate- gies and programs.  Some Road Authorities tend to consider maintenance and rehabilita- tion/upgrading as separate interventions.  Road Fund allocations to Road Authorities are inadequate for the maintenance needs of the networks with an over-reliance on the fuel levy compared to other road user charges.  Road funds are not always used for road maintenance.  Procurement and tendering processes are very time consuming and inef- ficient in Government road institutions and a lot less so in commercially oriented agencies.  The capacity of local consultants and contractors is still limited in a num- ber of countries.  Technical auditing of maintenance projects is generally carried out but the process is sometimes more superficial than rigorous. Auditing of de- velopment projects is seldom carried out. The positive impacts of the reforms so far, despite the challenges faced, indicate that the pursuit of the reforms promoted under the RMI is justified and beneficial. However, from the application of the CRM framework it can be concluded that for more agencies to realize the full benefits of commercialization, the following issues must be addressed in what may be termed “second generation roads agencies”:  More institutional autonomy should be given to Roads Authorities for accountability and improved efficiency through the amendment of exist- ing Road Acts, where necessary;  Road Boards should be constituted with majority private sector represen- tation and should introduce appropriate mechanisms for promoting good Board governance; including adherence to a code of ethics, and:  Board members should be adequately remunerated, should be required to be abreast of commercialized road management practices and to adhere to a code of ethics. 27 Road Management Policy– Road Agency Performance  Road Authorities should focus only on their “client” functions and should outsource all non-core functions.  Inter-agency (Road Authority/Road Fund) Procedures Agreements should guide the implementation of Road network management Plans and programs.  All Road Authorities should move to an integrated network management approach for managing their road networks, making good use of road as- set management systems.  Performance measurement of all Road Authorities should be undertaken on the basis of appropriately chosen performance indicators.  Where necessary, Road Authorities should contract appropriately experi- enced local consultants to take responsibility for the data collection, op- eration and management of Road Asset Management Systems.  The revenue collection base for Road funds should be widened and mechanisms should be put in place for automatic adjustment of road user charges in relation to road demand.  Road Authorities should undertake technical auditing of both mainte- nance and new construction/rehabilitation/upgrading projects at all stag- es of the project cycle.  Further, locally driven institutional reform is required in many road agencies in order to improve the efficiency and effectiveness of their road management operations based on implementation of the four Building Blocks with particular emphasis on the first – Creating Ownership – tak- ing due account of the examples of good practice that have emerged from this study. General findings and recommendations emanating from application of PAS framework The general findings derived from the application of the PAS framework are summarized as follows:  There is a general drive towards road asset management.  Member States have different perception about road asset management.  Risk related aspects of asset management (“thinking the unthinkable”) are underdeveloped. 28 Alternative Approaches for Assessing Road Agency Performance  There are funding gaps leading to vicious circle (no funds → no mainte- nance → degradation of the roads quality and value → need for more funds, etc.).  Maintenance is not politically attractive, leading to short term decisions instead of Life Cycle Costs.  There are generally fuzzy responsibilities between asset owner, asset man- ager and service provider.  Standard of performance measurement is moving from PAS 55 to ISO 55000. The general recommendations derived from the application of the PAS framework are summarized as follows:  The Member states’ top road agency management should demonstrate commitment with respect to asset management by: - Ensuring that an asset management policy that is aligned to ISO 55000 and compatible organizational objectives is established. - Ensuring that the asset management system requirements are in- tegrated into the organization’s business processes. - Ensuring that the resources (financial, software, people, equip- ment, training) for the asset management system are available. - Promoting cross cutting collaboration within the organization.  The Member states’ top road agency management should ensure that as- set management related risks are considered in the organization’s man- agement approach including contingency planning.  The Member states should perform self-assessment on annual basis and submit to ASANRA Secretariat by end of November each year.  Another asset management peer review process should be performed in 2017 to review progress made.  Road Fund Agency representatives should be permanent members of the Road Network Management and Financing Committee of ASANRA.  The impact of super single tires on regional road networks should be in- vestigated and appropriate policies recommended. 29 Road Management Policy– Road Agency Performance Comparison of outcome of assessments based on application of CRM and PAS frameworks A detailed comparison of the outcome of the assessment of road asset manage- ment capacity based on the application of the CRM and PAS frameworks for South Africa, Tanzania and Botswana is presented in Annexes A, B and C and is summarized in Table 9. Table 9–Comparison of outcome of selected country assessments by CRM & PAS frameworks Overall rating Country CRM PAS U P S G E U P S G E South Africa Tanzania Botswana Framework rating Legend: U=Unsatisfactory (0-20); Legend: 0 = Unsatisfactory, 0.1-1.0 = P=Poor (21-40); S =Satisfactory (41-60); Poor; 1.1-2.0 = Satisfactory; 2.1-3.0 = G=Good (61-80); E=Excellent (81-100) Good; 3.1–4.0=Excellent As is apparent from Table 9, both frameworks rate the three countries in a relative- ly, but not absolutely, similar manner. This difference in absolute terms is due to the dissimilar scales used in the frameworks in that the PAS framework includes a scale (Maturity Level Excellent: rating 3.1–4) surpassing the standard required to comply fully with the PAS 55 requirements considered to be achieved at the level Good of maturity (rating 2.1-3). Thus, if the scales of CRM and PAS were to be “standardized” by assuming that attainment of the PAS Maturity Level Good, which complies largely with all of the requirements of PAS 55, is rated as excellent, then this rating would be similar to the CRM scale of Excellent (rating of 81-100) which complies largely with all the requirements of the 4 BB of the RMI. Thus, with such standardization, both frameworks would produce the same results. Although both the CRM and PAS frameworks provide very similar results in terms of the overall rating of road agency capacity in road asset management, the under- lying implications of such an outcome are very different. This is due to the fact that the CRM framework addresses a much more extensive set of parameters that are likely to affect road agency asset management capacity than is the case with the PAS framework. While the PAS rating is determined by a set of PAS-specific pa- rameters, these parameters are influenced by additional parameters which are not included nor rated by the PAS approach but are taken into account in the CRM approach. For example, the PAS framework rates the South African road agency as 30 Alternative Approaches for Assessing Road Agency Performance “Good”. The reason for this rating is more than likely not only because the agency is compliant with the PAS requirements but, also, because it is, more broadly, also RMI compliant in terms of meeting generally the requirements of the four BB and as a result is rated Excellent by the CRM framework. In summary, the CRM framework evaluates all the key elements contained in the four RMI BB which, collectively, have been shown to influence the overall efficien- cy and effectiveness of a road agency’s performance. In contrast, the PAS frame- work focuses primarily on the asset management system aspects (BB4) within an agency which, unless complemented by other factors affecting road agency per- formance, is unlikely to provide a comprehensive assessment of road agency asset management capacity. In other words, the application of the PAS framework is a necessary, but not sufficient requirement, for fully assessing road agency asset management capacity. Scope for Improvement of CRM Framework From the outputs of previous diagnostic studies on road agency performance, it is apparent that without achievements of the four BB, lasting improvements in road asset management are unlikely. Thus, an “ideal” framework for assessing road agency capacity and performance should evaluate at least those factors included in the four BB. That being a necessity, the CRM framework would seem logically to provide a sound basis for refinement by incorporating the various elements of the PAS framework that focus more specifically, than does the CRM framework, on the asset management system aspects of a road agency’s operations. To this end, the conceptual framework in Figure 9 is proposed. The proposed amalgamated framework would also capture the organizational ca- pacity, external environment and organizational motivation components that af- fect road agency performance, as presented in the IOA framework in Figure 4. Nonetheless, it is recommended that this proposed new framework is closely re- viewed by all stakeholders, including ASANRA, ARMFA and road agencies in SSA countries before it is finalized. 31 Road Management Policy– Road Agency Performance Figure 9. Improvement of CRM evaluation framework (BB4) using complementary PAS elements Need for Performance Indicators The absence of an adequate framework of performance indicators (PIs) to measure and report upon on a continuing basis road agency performance in asset manage- ment is a drawback, which can have adverse consequences in terms of:  Uncertainty as to what progress is being made in achieving the reforms that have generally been accepted as a minimum requirement to facilitate commercialization of the road sub-sector  Inability of road agencies administrations to gauge the efficiency and ef- fectiveness of their operations  Tendency for policies and interventions to be made in a vacuum and to be re-active rather than pro-active In view of the above, there is a need to enhance the proposed amalgamated CRM- PAS framework indicated in Figure 9 by linking the outputs to appropriate PIs, as discussed in the next section. 32 5. Performance Measures and Indicators Introduction One way of measuring the progress being made by SSA countries towards the achievement of the more autonomous, accountable and business-like agencies envisaged by the RMI is through the use of appropriate Performance Indicators (PIs). As illustrated in Figure 10, the use of PIs has multiple dimensions and PIs can be used by stakeholders, such as governments, service providers and road us- ers, to measure the efficiency and effectiveness of the agency’s operations against targets or a baseline position, as a basis for taking adjusting action. Figure 10. Users of road sector performance indicators Source: Humplick and Paterson 1994. 33 Road Management Policy – Road Agency Performance With the use of PIs, monitoring and evaluation becomes more objective and is no longer rooted in personal judgment and narrative descriptions. Moreover, its use would enable road agencies to assess their performance and also enable peer com- parison between agencies in other countries – thereby providing a strong motiva- tion for improvement. Such self-assessment also has the advantage of enabling a fair and open assessment by removing the tendency of agencies to over- or under- rate themselves when being compared with similar agencies (Mihai et al, 2000). Existing Performance Indicators RMF Matrix The RMF Matrix of transport sector performance indicators has been developed as part of the SSATP/ARMFA Partnership program. This matrix consists of a number of performance indicators as shown in Table 10. Table 10. RMF matrix of performance indicators Area of Measurement Performance Indicator Road agency institutional and  Whether established or not operational framework  Whether a board exists  Whether the board has a private sector majority  Whether a road management system is used  % value of maintenance work contracted out Road network condition  % of primary roads in good or bad condition SSATP Transport Indicators SSATP also developed two sets of key transport sector performance indicators for Africa (SSATP, 2004) as follows: i. High level indicators. These have been developed for four focus areas, one of which is Road Network Management, in terms of:  Road condition: Percentage of the road network in good and fair condition.  Network standard appropriateness: Percentage of road carrying traffic less than the economic threshold of their type (50 vpd for a gravel road and 250 vpd for a paved road). ii. Secondary indicators. These have been developed for six sectors, in- cluding Transport Sector Management Sustainability and Road Transport in terms of the following relevant indicators. 34 Performance Measures and Indicators Transport Sector Management Sustainability a. Costs – 6 No. indicators b. Financial autonomy – 9 No. indicators c. Institutional Development/Governance – 2 No. indicators d. Transport sector impact – 2 No. indicators Road Transport a. Road network – 3 No. Indicators b. Road Usage – 4 No. Indicators c. Road Sub-Sector Management – 1 No. indicator d. Road Safety – 1 No. Indicator More recently, transport governance indicators have also been developed by SSATP in order to demonstrate the quality of governance in a particular country, sector or sub-sector (Christie, Smith and Conroy, 2012 – SSATP Working Paper No. 95). The scoring framework adopted in SSATP Working Paper No. 95 allows the indicators to be base-lined and then targets to be set as the basis for action. The scoring system is founded on a simple four-level RAG (red–red/amber– amber/green–green) rating approach, allowing assessment of performance along a continuum. An example of the RAG ratings is presented in Table 11. Table 11. Example of RAG rating system for a particular indicator Indicator Red Red/Amber Amber/Green Green Budget allocations More than 50% Less than 50% Less than 20% Sector financial based on reliable difference between difference be- difference be- ceilings and financial forecasts sector financial tween sector tween sector budget alloca- and aligned to ceilings and actual financial ceilings financial ceilings tions are con- priorities based on budget allocation (by and actual budget and actual budget sistent (less than objective criteria. amount). allocation. allocation. 10% difference). Both the RMF and SSATP indicators and the evaluation framework of the latter offer valuable inputs to the development of a measurement framework and related indicators for measuring and reporting progress in the efficiency and effectiveness of road agency performance as discussed in the next section. 35 Road Management Policy– Road Agency Performance Proposed Performance Indicators When compared with the more comprehensive set of performance indicators for assessing the efficiency and effectiveness of road administrations that are available (Humplick and Paterson, 1994), it is apparent that the RMF matrix and the SSATP Transport Governance Indicators should be expanded, where necessary, to ensure that the following areas of measurement are all linked to appropriate indicators: i. Performance Indicator 1. Commercialization of road management Measures the extent to which the road agency has progressed with the re- forms required to commercialize the management of roads. Indicators Indicate the extent to which progress has been made with regard to the var- ious policy, legislative, institutional, regulatory and administrative changes required to commercialize the management of roads. ii. Performance Indicator 2. Institutional effectiveness Measures the performance of the agency in terms of meeting institutional goals. Indicators Indicate the effectiveness of long-term network preservation strategies in- cluding resource lag, research and training. iii. Performance Indicator 3. Sectoral effectiveness Measures how closely the condition and operation of the network match the optimal state. Indicators Indicate technical and functional qualities as evidenced by the physical con- dition of the network and quality of service provided to road users. iv. Performance Indicator 4. Provision efficiency Measures the productivity and efficiency of the agency in providing the in- puts into the road system. 36 Performance Measures and Indicators Indicators Indicate how much resource was required to produce a unit of output in terms of such aspects as maintenance expenditure and staff productivity. v. Performance Indicator 5: Infrastructure provision Tracks the amount of the largely public asset, the size of the demand and the adequacy of the road system for the demand environment Indicators Provide information on demography and macro-economy, road network size and asset value, road network availability and utilization. The proposed framework for monitoring and evaluating the performance of road agencies is presented in Table 11 and includes the five measures described above and about 30 related PIs including relevant ones that have been retained from both the RMF matrix, the SSATP Transport Governance Indicators and the PAS evaluation framework. However, in order to harmonize these approaches, it will be necessary to adopt a similar rating scale. For ease of conformance, it is proposed to adopt the four level RAG system embedded in the SSATP Working Paper 95, as described above. Accordingly, it will be necessary to amend the CRM framework from a five level scale to a four level scale. This will make it possible to produce indicators that will be the outcome of a quantitative evaluation procedure and will fall into any one of the four rating scales. On this basis performance can be moni- tored both within and amongst road agencies. As indicated in Table 12, Performance Indicator 1 – Progress with Commercializa- tion, provides the means of tracking progress made by a road agency in the com- mercialization of road management practices. The rating system illustrated in Ta- ble 11 and embedded in Table 12, also allows the agency to gauge what reforms are required to take a particular feature of its performance from where it is towards where it should be. Performance Measures 2, 3 and 4 provide the means of track- ing progress in the efficiency and effectiveness of infrastructure provision. The outcome of the performance evaluation process can be presented in the form of a histogram (e.g. Figure 5) or a radar diagram (Figure 8). Data requirements Much of the data required to generate the PIs listed in Table 12 would be readily available either from national statistics (e.g. those pertaining to demography and 37 Road Management Policy– Road Agency Performance macro-economy) or from the regular monitoring procedures normally undertak- en by most road administrations and held in the database of a road or pavement management system. Some of the PIs are contained in the current RMF matrix (road condition) while a few are generated by external organizations (e.g. vehicle fleet size and motorization). Table 12. Framework of performance measures and indicators Performance Indicator 1. Measure of Progress with Commercialized Road Management Practices Indicator Building Block 1. Responsibility : Institutional Framework Indicates whether an RA has been established by an Act of Parliament as an institutionally separate, arms-length and func- tionally autonomous body to be responsible for service delivery and to be overseen by a separate Board. Red Red / Amber Amber / Green Green Autonomous/semi- Autonomous/semi- Roads Act provides for Roads Act provides for autonomous RA not autonomous RA estab- separate management independent Board to over- established lished through an Act of and funding of roads but see RA operations Parliament but no sepa- no independent Board to rate management and oversee RA operations. funding of roads. Similar approach for all elements of all Building Blocks as indicated in Table 2. Performance Indicator 2. Measure of Institutional Performance Effectiveness Indicator : Resource Lag Backlog maintenance by road category (% budget) Red Red / Amber Amber / Green Green A% B% C% D% Indicator : Research and Training Expenditure on research & training (% of total road expenditure) Red Red / Amber Amber / Green Green A% B% C% D% Performance Indicator 3. Measure of Sectoral Effectiveness Indicator : Preservation Effectiveness i. Pavement condition (% lane-km in Good/Fair/Poor condition. ii. Structures condition (% units In Good/Fair/Poor condition. Red Red / Amber Amber / Green Green Primary A% lane - km Primary B% lane - km Primary C% lane - km Primary D% lane - km Secondary A% lane -km Secondary B% lane - km Secondary C% lane - km Secondary D% lane - km Tertiary A% lane -km Tertiary B% lane - km Tertiary C% lane - km Tertiary D% lane - km Similar approach for all elements of all Building Blocks as indicated in Table 2. Indicator Axel load Control Degree of control over overloading Red Red / Amber Amber / Green Green A% B% C% D% Indicator : Road Safety Fatalities (no./Million VKT) Red Red / Amber Amber / Green Green A B C D 38 Performance Measures and Indicators Performance Indicator 4. Measure of Provision Efficiency Indicator : Output Efficiency Average maintenance cost/VKT Red Red / Amber Amber / Green Green A $/VKT B $/VKT C $/VKT D $/VKT Total expenditure/total agency staff ($/employee) Red Red / Amber Amber / Green Green A $/employee B $/employee C $/employee D $/employee Number maintenance contracts let/annum (km) Red Red / Amber Amber / Green Green A km B km C km D km Average time to procure contracts (days) Red Red / Amber Amber / Green Green A days B days C days D days Indicator : Provision mode Private supply participation: Contract expenditure/Total expenditure (%) Red Red / Amber Amber / Green Green A% B% C% D% Contract expenditure/Agency force a/c expenditure (%) Red Red / Amber Amber / Green Green A% B% C% D% Performance Indicator 5: Measure of Infrastructure Provision Indicator Units Comments Demography & macro- economy Gross Domestic Product US $M Significant travel demand determinant. Fiscal cost (roads) % Total road expenditure as % GDP. Road network size Road length km By functional class/type. Descriptor of the road system. Bridges m By functional class and type. Tunnels m By functional class and type. Road asset value Asset value $M Current value by component. Allows asset value to be tracked. Road network availability Road density km/1000 inhabitants Provides an indicator of accessibility to population. Road coverage km/100 km2 Provides an indicator of accessibility to land. Road users Km/100 Vehicle fleet size Million veh. By category. Is an important travel demand factor. Motorisation Veh /1000 inhabitants Harmonization of Performance Indicators In order to ensure that the PIs proposed in Table 12 are used in a consistent man- ner in all SSA countries, it is important that they are defined and specified in a unique manner and that the supporting data for calculating them is collected in a harmonized manner. In particular, to focus on performance evaluation, many indicators proposed for the measure of infrastructure provision should be trans- 39 Road Management Policy– Road Agency Performance formed to fit within the format of the RAG framework where actual performance is measured against targets. This will require the preparation of a specification for each PI as illustrated in Table 13. Table 13 – Illustrative example of a PI specification Performance Measure Infrastructure provision Indicator Code RD.P Indicator Name Road density Purpose Measure of accessibility Desired Goal Improved national accessibility Indicator Description Kilometer length of total road network per 1,000 people. The total road length in- cludes (a) motorways/freeways, (b) highways, main or national roads, (c) secondary or regional roads, (d) urban roads, (e) rural roads (all classified or gazetted roads). Indicator Definition RD.P = L/(P/1000) Illustrative Case In 2007, the total road length of country C was 8,700 km and its total population was 1.7 million people. Illustrative calculation L = 8,700 P = 1,700,000 RD = 8,700/(1,700,000/1,000) = 5.1 The approach outlined above should be undertaken for each indicator. However, it is outside the scope of this study to define all the indicators to this level of detail as the final selection of indicators and buy-in from stakeholders should be part of an implementation strategy aimed at engendering ownership of the performance monitoring and evaluation framework outlined in this section. Integrating Performance Measures and Indicators into a Road Agency The following actions would be required to ensure successful integration of the PIs into the road agency management processes. These include (TRB, 2006):  Engage all stakeholders to achieve buy-in and commitment to use the measures  Integrate the performance measures into existing processes and decision- making forums  Agree on clear ownership and responsibility for each measure and associ- ated data and tools  Identify needs for additional data collection, data management, and ana- lytic tools to support the selected measures and indicators  Design communication tools with formats appropriate to rget audiences 40 Performance Measures and Indicators  Document measure definitions and procedures  Estimate the cost of operating and maintaining the measurement frame- work and ensure that there is a matching budget. 41 6. Findings, Conclusions and Recommendations The following is a summary of the conclusions and recommendations arising from the review of the CRM and PAS frameworks for assessing road agency capacity and performance in road asset management. Main Findings and Conclusions Comparison of CRM and PAS approaches for assessing road agency asset management capacity i. The PAS and CRM frameworks for assessing road agency capacity in asset management have been developed in quite different ways, against quite different environmental and cultural backgrounds and are quite differ- ently focused. ii. The CRM framework is the product of the RMI concept of commerciali- zation of road management, which was initiated in response to chronic shortcomings in the ability of road agencies to manage their road net- work in an efficient and effective manner. The framework addresses key aspects of road asset management that are perceived as being of critical importance for achieving proficiency in road asset management. The framework considers that reforms in the road sector, in line with the commercialization concept, are a necessary pre-requisite for achieving sustainable improvements in road asset management. Moreover, follow- up studies have shown that compliance with the requirements of the commercialization concept have generally resulted in improved condi- tions on the ground. iii. The PAS framework is an internationally recognized benchmark for road asset management developed by the UK Institute of Asset Management and the British Standard Institute in response to demand from the indus- try in Europe for a standard for asset management. This standard is based on compliance with an international specification for the optimized management of physical assets and the main focus is on management of physical assets largely through the use of an asset management system in- cluding a supporting asset management strategy and related asset man- 43 Road Management Policy– Road Agency Performance agement objectives and plans. Its application, so far, has been in relatively mature and developed road agencies in developed countries. iv. In view of its relatively narrow focus, the PAS framework provides a nec- essary but limited basis for undertaking an overall evaluation of a road agency’s capacity and related performance in asset management. Indeed, many of the problems with road asset management are caused by funda- mental institutional and organizational weaknesses which are often inter- related. Thus, a good road asset management system will make little dif- ference to the quality of roads in a country if the level of funding is erratic or the quality of staff inadequate. Comparison of findings emanating from application of the CRM and PAS frameworks for assessing road agency capacity in road asset management i. The CRM framework has probed in some detail the factors associated with the achievement of the RMI commercialization concept as included in its four inter-related Building Blocks (BB). Thus, it has pin-pointed fairly specifically the extent of an agency’s non-compliance with, for ex- ample, those elements of organizational structure, planning, management and financing of roads that critically affect the ability of the agency to ful- fil its mandate in an efficient and effective manner. ii. As a result of the approach embedded in the CRM framework, it has been possible to not only indicate areas of weakness but, also, how to address them in terms of the required improvements/reforms in line with the four BB. iii. The PAS framework has probed quite extensively those elements of road agency operations related to the establishment and operation of a road asset management system, including asset management strategy, asset management objectives and asset management plans. Ironically, one of the findings of the CRM study was that "Most road agencies are unable to operate their Road Asset Management Systems to produce reliable out- puts in terms of optimal network strategies and programs”. iv. As a result of the approach embedded in the PAS framework, it has been possible to pinpoint what needs to be done to achieve the PAS specifica- tion for road asset management, but not how to go about achieving this, which must be determined by the agency in accordance with its assessed needs. For example, maintenance funding gaps have been identified as a major challenge faced by road agencies but the root cause of the problem 44 Findings, Conclusions and Recommendations in most countries – insufficient, erratic, unsustainable funding – and the manner of addressing the challenge, as detailed in the RMI commercializa- tion concept, and the need to attain the elements of BB3: Providing Stable Financing, have not been identified as key issues. v. In summary it can be concluded that the CRM framework provides a broader approach to determining road agency capacity in road asset man- agement. The framework addresses not just the asset management system aspects, but also other related aspects associated with the institutional ar- rangements, planning, management and financing aspects of roads, all of which critically affect road agency asset management capacity. In contrast, the PAS framework is more narrowly focused on the road asset manage- ment system aspects and much less so on other factors that affect road agency asset management capacity. Comparison of outcome of assessments based on application of CRM and PAS frameworks Both the CRM and PAS frameworks provide very similar results in terms of the overall rating of road agency capacity in road asset management. However, the underlying implications of such an outcome are very different in that the CRM framework addresses a much more extensive set of parameters that are likely to affect road agency asset management capacity than is the case with the PAS framework. Scope for improvement of CRM framework The CRM and PAS frameworks offer complementarity in the assessment of road agency capacity in road asset management in that CRM as broader framework can be refined by incorporating the more specifically focused asset management system aspects of the PAS framework. Performance indicators i. Both the CRM and PAS evaluation outcomes are quantitatively based and can be used as performance indicators that measure progress in commer- cialized road management. ii. The RMF and SSATP indicators proposed in Working Paper No. 95: Transport Governance Indicators for Sub-Saharan Africa, together with the CRM and PAS indicators, offer valuable inputs to the development of a measurement framework and related indicators for measuring and re- 45 Road Management Policy– Road Agency Performance porting progress in the efficiency and effectiveness of a road agency. However, they need to be supplemented with other indicators to cover other aspects of road agency performance including institutional effec- tiveness, sectoral effectiveness and provision efficiency. Moreover, the in- dicators will need to be defined and specified in a unique manner and the supporting data for calculating them will need to be harmonized to en- sure consistency in their use. Main recommendations The following key recommendations emerge from this study. i. The CRM and PAS evaluation frameworks should be amalgamated so as to produce a consolidated framework for evaluating road agency capacity. Such a framework would incorporate the strengths of the individual frameworks, including all four BB embedded in the commercialization concept. In so doing, the various elements of both frameworks should first be reviewed by key stakeholders, including ASANRA, AGEPAR and ARMFA, to engender full buy-in and ownership of the new framework. ii. The RMF and relevant SSATP indicators proposed in Working Paper No. 95: Transport Governance Indicators for Sub-Saharan Africa, together with the CRM and PAS indicators should be expanded in a harmonized man- ner to ensure that the following performance measures are all catered for. a. Performance Indicator 1: Measures Progress with commercialization b. Performance Indicator 2: Measures Institutional effectiveness c. Performance Indicator 3: Measures Sectoral effectiveness d. Performance Indicator 4: Measures Provision efficiency e. Performance Indicator 5: Measures Infrastructure provision 46 Annexes Annex A. Outcome of assessment of road asset management capacity in South Africa based on CRM and PAS evaluation frameworks Table A. Outcome of assessment of road asset management capacity in South Africa based on CRM and PAS frameworks Outcome from Application of CRM Framework U P S G E BB1 Institutional framework Organization BB2 Oversight management BB3 Secure and stable financing User pays principle BB4 Degree of autonomy Management structure & op procedures Human resources Financial management Management information systems Procurement and tender procedures Quality control Overall rating 94.8 Legend: U=Unsatisfactory (0-20); P=Poor (21-40); S =Satisfactory (41-60); G=Good (61-80); E=Excellent (81-100) Outcome From Application of PAS Framework U P S G E BB1 General requirements Asset management policy BB2 Asset management strategy, objectives and plans BB3 Asset management enablers and controls Implementation of asset management plans BB4 Performance assessment and review Management review Overall rating 2.7 Legend: 0 = Unsatisfactory, 0.1 - 1.0 = Poor; 1.1 - 2.0 = Satisfactory; 2.1 - 3.0 = Good; 3.1 – 4.0=Excellent 47 Annex B. Outcome of assessment of road asset management capacity in Tanzania based on CRM and PAS evaluation frameworks Table B. Outcome of assessment of road asset management capacity in Tanzania based on CRM and PAS frameworks Outcome from Application of CRM Framework U P S G E BB1 Institutional framework Organization BB2 Oversight management BB3 Secure and stable financing User pays principle BB4 Degree of autonomy Management structure & op procedures Human resources Financial management Management information systems Procurement and tender procedures Quality control Overall rating 71.4 Legend: U=Unsatisfactory (0-20); P=Poor (21-40); S =Satisfactory (41-60); G=Good (61-80); E=Excellent (81-100) Outcome from Application of PAS Framework U P S G E BB1 General requirements Asset management policy BB2 Asset management strategy, objectives and plans BB3 Asset management enablers and controls Implementation of asset management plans BB4 Performance assessment and review Management review Overall rating 1.2 Legend: 0 = Unsatisfactory, 0.1 - 1.0 = Poor; 1.1 - 2.0 = Satisfactory; 2.1 - 3.0 = Good; 3.1 – 4.0=Excellent 49 Annex C. Outcome of assessment of road asset management capacity in Botswana based on CRM and PAS evaluation frameworks Table C. Outcome of assessment of road asset management capacity in Botswana based on CRM and PAS frameworks Outcome from Application of CRM Framework U P S G E BB1 Institutional framework Organization BB2 Oversight management BB3 Secure and stable financing User pays principle BB4 Degree of autonomy Management structure & op procedures Human resources Financial management Management information systems Procurement and tender procedures Quality control Overall rating 45.1 Legend: U=Unsatisfactory (0-20); P=Poor (21-40); S =Satisfactory (41-60); G=Good (61-80); E=Excellent (81-100) Outcome from Application of PAS Framework U P S G E BB1 General requirements Asset management policy BB2 Asset management strategy, objectives and plans BB3 Asset management enablers and controls Implementation of asset management plans BB4 Performance assessment and review Performance review Overall rating 0.7 Legend: 0 = Unsatisfactory, 0.1 - 1.0 = Poor; 1.1 - 2.0 = Satisfactory; 2.1 - 3.0 = Good; 3.1 – 4.0=Excellent 51 References and General Bibliography 1. Benmaamar, M. (2006). Financing of road maintenance in Sub-Saharan Africa. Re- forms and progress towards second generation road funds. SSATP Discussion Paper No. 6. The World Bank, Washington, D.C. 2. Brooks, D. M., Robinson R. and O’Sullivan K. P. (1989). Priorities in improving road maintenance overseas: a check-list for project assessment. Proceedings of the Institu- tion of Civil Engineers, Part 1. 86 (Dec), 1129-1141. 3. Smith, D., Christie, A., and Conroy K. (2013). Transport Governance Indicators for Sub-Saharan Africa. SSATP Working Paper No. 95. The World Bank, Washington. 4. Gwilliam, K., Foster, V., Archondo-Callao, R., Briceno-Garmendia, Nogales, A. and Sethi K. (2008). Africa Infrastructure Country Diagnostic: Roads in Sub-Saharan Africa. The World Bank, Washington, D.C. 5. Gwilliam, K., Foster ,V., Archondo-Callao, R., Briceno-Garmendia, C. and Sethi, K. (1998). Africa Infrastructure Country Diagnostic: Roads in Sub-Saharan Africa. SSATP Summary Background Paper No. 14. The World Bank. Washington, D.C. 6. Heggie, I.G. (1995). Commercializing Africa’s Roads: Transforming the Role of the Public Sector. Transport Reviews 15 (2): 167-84. 7. Heggie, I. G. and Kerali, H. (2000). Checklist for Examining Road Agency Perfor- mance. Sourcebook on Institutional Development for Utilities and Infrastructure. Report prepared for Department for International Development (DFID), London. 8. Heggie, I.G. and Vickers, P. (1998). Commercial management and financing of roads. World Bank Technical Paper No 409. The World Bank, Washington, D.C. 9. Humplick, F. and Paterson W. D. O. (1994). Framework of Performance Indicators for Managing Road Infrastructure and Pavements. Conf. Proc. Vol. 1. Third Int. Conf. on Managing Pavements, San Antonio, Texas, May 22-26, 1994. Washington, D.C. Trans- portation Record Board, National Academy Press, 1994. 10. Lusthaus, C., Adrien, M. H., Anderson, G., & Carden, F. (1999). Enhancing organiza- tional performance a toolbox for self-assessment. Ottawa, Canada: International De- velopment Research Centre, Ottawa. 53 Road Management Policy– Road Agency Performance 11. Mihai, F., Binning N. and Dowling L. A framework for Assessing Asset Management Performance in Australia. Main Roads, Western Australia. 12. Pinard, M. I. (2012). Progress on Commercialized Road management in Sub-Saharan Africa. SSATP Working Paper No. 92. The World Bank, Washington, D.C. 13. Queiroz, C. and Kerali H. (2010). A Review of Institutional Arrangements for Road Asset Management: Lessons for the Developing World. Transport Paper TP-32, The World Bank, Washington, D.C. 14. Robinson, R. (2008). Restructuring Road Institutions, Finance and Management. Vol. 1: Concepts and Principles. University of Birmingham. 15. Robinson, R., Danielson, U. and Snaith M. (1998). Road maintenance management: concepts and systems. Basingstoke and London: Macmillan. 16. Southern Africa Development Community (SADC). (1999). Protocol on Transport, Communications and Meteorology, Gaborone. 17. Southern Africa Transport and Communications Technical Unit (SATCC) (1999). Model Legislative Provisions: Road Network Management and Financing, Maputo. 18. Transportation Research Board. (2006). Performance Measures and Targets for Trans- portation Asset Management. National Cooperative Highway Research Program (NCHRP) Report 551, Washington, D.C. 19. Stankevich, N., Guillossou, J.N. (2008). Assessment of road funds in South Asia region. Transport Notes series TRN 37. The World Bank, Washington, D.C. 54