MONITORING AND EVALUATION CAPACITIES IN GHANA A DIAGNOSIS AND PROPOSED ACTION PLAN The World Bank January 2000 L:\GhanaECD\Ghana Diagnosis and Action Plan 1 ABBREVIATIONS AND ACRONYMS AEA African Evaluation Association AfrDB African Development Bank CDF Comprehensive Development Framework CIDA Canadian International Development Agency CMA Central Management Agencies COSCO Committee on Strategy Coordination CSPIP Civil Service Performance Improvement Programme CWIQ Core Welfare Indicators Questionnaire DFID Department for International Development (UK) EU European Union FAO Food and Agriculture Organization GIMPA Ghana Institute of Management and Public Administration GOG Government of Ghana GSS Ghana Statistical Service GTZ German Technical Cooperation JAI Joint Africa Institute M&E Monitoring and Evaluation MDA Ministries, Departments and Agencies MOF Ministry of Finance MOFA Ministry of Food and Agriculture MRT Ministry of Roads and Transport MTEF Medium Term Expenditure Framework NDPC National Development Planning Agency NGO Non-Government Organization NIRP National Institutional Renewal Programme NOC National Overview Committee OCS Operational Core Support (World Bank) OED Operations Evaluation Department (World Bank) OHCS Office of the Head of the Civil Service PMG Policy Management Group PPMED Policy, Planning, Monitoring and Evaluation Department PSC Public Services Commission PSM Public Sector Management PSMRP Public Sector Management Reform Programme PUFMARP Public Financial Management Reform Programme SAPRI Structural Adjustment Participatory Review Initiative SEC State Enterprises Commission SOE State Owned Enterprises TA Technical Assistance UNDP United Nations Development Program WBI World Bank Institute 2 CONTENTS Summary i The Diagnosis 1 Step 1: Identify Key Ministries and Other Bodies 1 Step 2: Diagnose Public Sector Incentives, Ethics 2 Step 3: Identify What Influences Budget and Line Management Decisions 3 Step 4: Define The Extent of Influence of M&E 5 Step 5: Identify M&E Activities of GOG 6 Step 6: Identify M&E Activities/Roles of Development Assistance Agencies 8 Step 7: Identify Major Public Sector Reforms in Recent Years 8 Step 8: Demand for and Supply of M&E 10 Options for Developing M&E Capacities 13 1. GOG Policy Statement on M&E/Performance Measurement 13 2. Development of An Effective M&E Framework Within GOG 13 3. Identification of M&E Champion(s) Within GOG 14 4. Deepening of the Performance Orientation Within the MTEF 15 5. Review the Structure, Functions, Resources and Practices of PPMEDs 16 6. `Ministry Mentoring' 18 7. Skills Enhancement Via Provision of M&E Training 21 8. Support Establishment of a Ghanaian Evaluation Forum/Network 22 9. Creation of an Evaluation Foundation 23 10.Support for Civil Society Involvement in Performance Measurement 25 Conclusions 28 Annexes A. An Action Plan for the Development of Ghanaian M&E Capacities 30 B. Reviews of PPMED Functions in Selected GOG Ministries Generic Terms of Reference 35 C. Training in M&E 37 i SUMMARY World Bank missions to Ghana were undertaken in June and October 1999 as part of a diagnosis of M&E capacities and related development options in Ghana. The missions were in response to a request from the government of Ghana (GOG) for assistance in strengthening its M&E functions. The government has an ambitious program of reform to improve public sector management reforms planned or underway include a significant downsizing of the public sector, an ongoing review of its functions and structures, a medium-term expenditure framework (MTEF), performance plans for ministries, departments and agencies, performance agreements for senior civil servants, and extensive beneficiary surveys. These reforms are being supported by, inter alia, a loan recently agreed by the Bank's Board to support the government's public sector management reform programme (PSMRP). Work to build capacities for a range of types of performance measurement (M&E broadly defined) has the potential to provide powerful support to these public sector reforms. The Bank's country department is actively supporting these reform efforts, including the development of M&E capacities. An important feature of Ghana is that it is a pilot country for the Comprehensive Development Framework (CDF). One of the pillars of the CDF is results and accountability pilot countries are strongly encouraged to embrace a results- based management framework and to fully embrace dialogue with civil society. The building of M&E capacities is a key vehicle to help ensure governments have the ability to measure and report on development effectiveness, to identify appropriate policy responses, and to engage civil society and NGOs more fully in these processes. Such national capacity-building also provides a counterpart to the M&E capacities of donors, which have been the focal point for results- orientation in the past. The CDF emphasis on poverty reduction will be reinforced by the government's preparation of a Poverty Reduction Strategy Paper. This paper will map out the obstacles to poverty-reduction and the proposed policy responses to them. It will identify intermediate and final outcome indicators, the role of civil society in monitoring these outcomes, and the capacity building and technical assistance necessary to support the strategy. The M&E capacity building which is proposed in this paper will provide key support for the poverty-reduction strategy. A preliminary diagnosis of the Ghanaian government's M&E capacities was prepared in July 1999. This identified a number of options which could be pursued to help build national capacities, and these options were the subject of discussion and debate at a workshop, for senior officials and other donors, in Accra in October 1999. ii There was strong agreement among senior officials at the workshop that M&E is an important element of public sector reform and a priority for the government not as a new or additional public sector reform initiative, but rather to provide a deepening and supportive element to reforms already underway. Thus there was a strong appreciation that sound M&E capacities are necessary to measure development effectiveness and to achieve a performance culture in the civil service. The workshop discussions identified a range of priority actions for developing the government's M&E capacities, and these are incorporated in the proposed action plan outlined in Annex A. Note that most but not all of the options identified below are included in the proposed action plan some options would be premature but all of the options have benefited from the workshop comments and discussion. During the October mission a separate workshop was conducted for representatives of NGOs and think-tanks, parliamentarians and others from civil society. The workshop examined good-practice examples from Ghana and other developing countries of civil society involvement in assessing public sector performance. The key part of the workshop was the brainstorming group sessions, in which participants were asked to identify the role that civil society wants to play in assessing public sector performance, to identify the main impediments to playing this role, and to outline priority steps for achieving deeper civil society engagement in this area. This workshop helped to continue a dialogue between donors and civil society, and was also useful in identifying several ways in which donors could support a more active role of civil society in Ghana. The proposed action plan presented at the end of this paper encompasses ways in which civil society's capacities for assessing public sector performance could be developed and supported. Based on the experience of the Bank and other donors, it is clear that some options and some combinations of options will be particularly important if the development of M&E capacities in Ghana is to be pursued in a sustainable manner. The ten options are briefly outlined below; they overlap to some extent. The options are: 1. GOG policy statement on M&E/performance measurement this would ideally be issued by a senior minister, such as the President or Vice- President, to ensure the credibility and profile of the statement 2. development of an effective M&E framework within GOG to set out clear responsibilities for different actors within GOG, i.e., who is expected to do what, why, when, and how. A framework would encompass the planning, iii conduct, quality, reporting and uses of M&E findings, at the national, sectoral and regional/district levels 3. identification of M&E champion(s) within GOG a ministry or high-level group or committee which could be a national leader and supporter of M&E capacity development 4. deepening of the performance orientation within the medium-term expenditure framework (MTEF) in particular, by ensuring that the training provided to ministries in support of the MTEF provides the necessary focus on government objectives, on measures of success, and on the strengths and weaknesses of different M&E tools 5. review of the structure, functions, resources and practices of selected ministries' Policy, Planning, M&E Departments (PPMEDs) and of related statistics units PPMEDs are meant to play a key role in M&E, and have the potential to do so. However, they appear to suffer from low capacities and to be engaged in some redundant activities 6. `Ministry mentoring' this would entail intensive long-term donor support for selected ministries' M&E activities, in order to establish and to demonstrate to others a range of good-practice approaches to M&E 7. skills enhancement via provision of M&E training this would need to be tightly targeted towards priority ministries and to some extent towards civil society, and would also need to be managed carefully to ensure the various types of M&E training provided are both appropriate and cost-effective 8. support the establishment of a Ghanaian evaluation forum/network this would help evaluators within and outside GOG to share insights into common problems and good practice approaches 9. creation of an evaluation foundation to identify and commission selected strategic evaluations i.e., major evaluations of a small number of strategic GOG programs or activities. The steering committee for the foundation might include GOG representatives, including senior officials, and representatives of think-tanks, NGOs and the private sector 10.donor support for capacity-building of NGOs, think tanks and the parliament. For NGOs and think tanks, donor support might appropriately extend beyond various types of M&E and the review of public sector performance, to encompass areas such as communication, policy advocacy, fundraising and budget analysis. Some of these areas would need to be addressed if NGOs and think tanks are to be able to contribute more fully to the assessment of public sector performance. The paper identifies the pros and cons of each of these options, and then reaches a conclusion about the appropriateness of pursuing the option now. There is a trade-off between the synergies from pursuing all ten options simultaneously and the burden of tackling them in that manner. There are issues of what options are related and can be clustered, and what would be a desirable sequencing. iv One useful strategy would be to develop good practice examples as demonstration pilots by targeting support for M&E capacity building to several ministries/agencies where there are good prospects for early success. This would be a particularly powerful approach where the senior management of the ministries are strong believers in the benefits of M&E. This could be pursued by the combination of several options (such as options 1 and 3 in the short-term, followed soon after by options 4, 5, 6, 7, 9 and 10) as the first phase of a longer- term approach which would desirably entail applying the options more generally across GOG ministries, including eventually the adoption of option 2. The wider application of these options would definitely benefit from the strong support of a central ministry or committee as a GOG-wide championindeed, this could be viewed as a necessary step if GOG wants to move forward decisively with M&E capacity building. A cut-down version of this overall approach would also be sustainable. This could encompass selected 'champion' line ministries, together with option 1 in the short-term, followed soon after by options 4, 5 and 10 plus provision of a range of M&E training (a more modest version of option 7). The proposed action plan includes all options with the exception of option 2, which is considered premature. It will be important to gauge the views of GOG particularly the views of potential champions of M&E on the proposed action plan. 1 THE DIAGNOSIS This paper provides a diagnosis of M&E capacities and development options in Ghana. It is based on a mission undertaken by Bank staff and a consultant in June 1999, a second mission in October 1999, a desk review of a number of documents on Ghanaian governance issues which have been prepared by the World Bank and by the Government of Ghana (GOG), plus several workshops with senior officials and with representatives of civil society. The paper follows the framework developed in Evaluation Capacity Development A Diagnostic Guide and Action Framework.1 Step 1: Identify key ministries and other bodies, and their formal/stated relationships Key central management agencies (ministries) include the following: · Ministry of Finance (MOF) including the Controller and Accountant General, which oversights budget aggregates and is responsible for monitoring the expenditure of other ministries · National Development Planning Commission (NDPC) responsible for national planning and coordinating the planning work of sector ministries · Public Services Commission (PSC), responsible inter alia for incomes policy reform · Office of the Head of the Civil Service (OHCS) responsible inter alia for the civil service performance improvement programme (CSPIP) · the State Enterprises Commission (SEC), which oversights the performance of state-owned enterprises. Sector ministries are responsible for identifying sectoral policy priorities and for oversighting the performance of government departments and agencies, including subvented agencies which provide services directly to citizens such as the Ghana education service and the police service. There are several key coordination bodies which have substantive responsibilities in GOG's public sector reform program. These include: · the National Institutional Renewal Programme (NIRP) secretariat, whose role is to push, monitor and evaluate the reform process. The secretariat is located within the Office of the Vice-President and reports to the National Overview Committee NOC comprises a number of ministers, chief directors of ministries, and a representative of the private sector. NOC members comprise all the top managers of the various public sector reform initiatives · the Policy Management Group (PMG), recently set up by the President to monitor and control policy processes, to ensure that GOG's policy priorities 1Keith Mackay, The World Bank, Operations Evaluation Department Working Paper No 6, January 1999. 2 are actually implemented, to assess the development impact of government policies, and to develop policy options. Assessments of government performance will be central to the work of this group. The PMG currently comprises 11 political appointees, each of ministerial rank, supported by a number of technical policy analysts · the PUFMARP (Public Financial Management Reform Programme) secretariat, located within the MOF. This body oversights a number of related reforms of the budget system, including the MTEF and the public expenditure management system · the Committee on Strategy Coordination (COSCO), recently set up by the Vice-President to oversight the objectives and performance of all ministries. The COSCO will comprise six key central agencies. Step 2: Diagnose public sector incentives, ethics, possible corruption. Determine what incentives and rules systems shape decision-making Sector ministers and their ministries enjoy considerable autonomy in determining sector priorities. Nevertheless, at the administrative level, there exist considerable central (PSC) controls on staffing levels and classifications. Budget allocations are strictly controlled and monitored by the MOF, although greater flexibility is in prospect with the progressive implementation of a medium-term expenditure framework approach; however, sector ministries face considerable uncertainties over the timing of receipt of their budget allocations due to macroeconomic management difficulties including the forecasting of tax revenues and this considerably complicates the management of ministries (and significantly reduces their willingness to be accountable for performance). There is little evidence of a widespread performance culture within GOG, although the major reforms now underway reveal a strong desire to improve the performance of the public sector. There appear to be few individual rewards for good performance (other than promotion), and few if any sanctions for poor performance. The civil service heads of many ministries are political appointees. It was suggested to Bank staff that corruption is extensive but of a generally low level, with the possible exception of the roads sector, where corruption may be more serious. GOG accepts that the performance and motivation of public servants is poor, and that this has resulted in unsatisfactory standards of service delivery. GOG has embarked on a series of public sector reforms to reduce the size, and to improve the functioning and performance, of the public sector. 3 Step 3: Identify what influences budget decision-making and line management decisions. Identify actual roles and extent of autonomy of central and line ministries GOG has clearly stated its commitment to sound economic management and to an extensive program of public sector rationalization and downsizing, which is now underway. Budget aggregates contain only a small amount of discretionary expenditure each year, and information on sector performance apparently has little influence on resource allocation decisions. The bulk of spending is determined by longer-term commitments, both domestic and donor-funded. MOF plays a central role in forecasting and advising on budget aggregates, and provides advice on inter-sectoral allocations to the cabinet during the budget process. However, MOF has a lack of budget policy analysts and does not play the type of role undertaken by finance ministries in developed countries, for example, which provide an alternative source of advice on intra-sectoral policy priorities. Sector ministers and their ministries, together with donors, play the dominant role in determining sector priorities, and MOF is content with this situation. MOF's main value-added in terms of performance management appears to be its work on physical monitoring of project implementation2 essentially for auditing purposes. An important development in Ghana has been the recent adoption of a medium- term expenditure framework (MTEF) under the aegis of MOF/PUFMARP. The MTEF entails three-year forward estimates, consolidated spending envelopes with four line items, and the specification of outputs to be achieved over the coming year this MTEF approach constitutes a type of performance budgeting. The MTEF commenced in FY99 and is still being developed. It is intended to include ministry mission statements (objectives), strategic plans and an explicit focus on the costing of activities and policies. One MTEF element which has been identified by MOF/PUFMARP officials for further development is the clear specification of expected and actual outputs of ministries, departments and agencies. Adoption of an MTEF framework is an important innovation and suggests a stronger and more explicit emphasis on GOG's sectoral priorities, and on the actual performance of ministries in achieving these priorities including on the accountability that is the counterpart of the greater predictability of sectoral allocations. A key challenge will be to ensure that a system is in place to ensure the timely availability of M&E findings to support the determination of inter- sectoral and intra-sectoral priorities. 2 This includes monitoring of project construction and also of the delivery of non-physical projects, such as the outputs or targets of workshops and training. In this context, the Minister for Finance has initiated value-for-money reviews of some 22 contracts. MOF has a contract, with the department of architecture of the University of Science and Technology in Kumasi, to document and monitor all 2,000 on-going projects which comprise the public investment program. 4 The health sector provides a good practice approach to donor funding and to broad reviews of overall sector performance. Some donor funds for this sector are pooled, and there is an agreed approach to joint annual GOG-donor reviews of this sector two such reviews have been conducted so far. A pooled approach means that donors give up the right to select which projects to finance, in exchange for having a voice in the process of developing sectoral strategy and allocating resources. The World Bank's support to the health sector is to be closely linked to performance specifically to progress in achieving sectoral development targets. Some 20 performance indicators have been jointly agreed for the health sector. The actual role of the NDPC, the central national planning body, is unclear. It is a small organization (32 staff) and its existence is mandated by the Constitution. It prepared the national development plan (Vision 2020) in 1997, relying heavily on inputs from the sector ministries. The national plan also contains an M&E system which sets out the M&E responsibilities of ministries (essentially relating to monitoring the implementation of activities) and the required flows of monitoring information to central agencies such as NDPC. However, there is no clear evidence that M&E findings have a significant influence on budget decision-making in terms of inter-sectoral or intra- sectoral allocations of resources (see discussion under step 4). At the June workshop for officials NDPC was criticized by senior officials of other ministries as adding little value to policy or planning deliberations, and as having uncertain leadership and direction. The NIRP secretariat, as part of the public sector reform program it is managing, is currently reviewing the mandate and functions of the NDPC and of other central ministries. The apparently weak role of the NDPC and the narrow role undertaken by MOF imply a lack of alternative domestic sources of policy advice to those provided by sector ministries, particularly in relation to the intra-sectoral allocation of resources. The apparent weaknesses in both organizations suggest a need to enhance their skills, expertise and orientation. There may be scope for NDPC to be more active, and to perform better, in undertaking strategic reviews/assessments, including sector reviews, and in assessing overall development progress since Vision 2020 was prepared. And there seems to be significant scope for MOF to broaden its perspective and shift its emphasis beyond processes/activities, so that it focuses much more on the outputs, outcomes and impact of government activities. 5 Step 4: Define the extent of influence of M&E on budget decision-making and line management There is no evidence that M&E findings have any significant influence on budget decision-making. Most sector ministries contain Policy, Planning, M&E Departments (PPMEDs), and all are required by law to do so. PPMEDs typically play a central role in collecting and coordinating data within the ministry and from the departments and agencies which the ministry oversights, and in planning and coordinating the preparation of budget bids from the various parts of the ministry. The size and functions of PPMEDs appear to vary widely and, by all accounts, so does their performance. The Bank missions met with officials from only a small number of PPMEDs, but gained the impression from them and others that their focus tends to be more bottom-up i.e., project and activity based, and principally concerned with the monitoring of financial inputs and of activities. PPMEDs also appear to have some capacity to collect primary data. Some ministries contain a separate statistics unit responsible for collecting data on ministry and sector performance; in other ministries these statistical functions are performed by the PPMED. PPMEDs' ability to make good use of these data or to support ministry management's use of them, or to take a sector-wide approach, is unclear. A number of donor representatives and officials noted a need to strengthen PPMEDs, given their actual or potential central role within sector ministries, and given sector ministries' key role in sector planning and management. Indeed, it appears that one omission in the long and impressive list of GOG public sector reforms now underway is a clarification of the role and functions of PPMEDs, as well as a strengthening of their capacities. This also bears on the interface between sector ministries and central ministries, particularly the NDPC which as noted above appears to be substantially under-performing its central planning functions. A related issue which was raised several times during the mission is the apparently weak links between sector ministries, reflecting the vertical boundaries between them. The need for greater coordination and collaboration between sector ministries is likely to emerge more acutely in future, as a result of the application of the Comprehensive Development Framework (CDF) approach between GOG and donors this approach highlights the importance of inter- sectoral issues to the achievement of each sector's objectives. The health sector reviews, described briefly earlier, embody a number of good- practice principles. These include close GOG-donor collaboration and review, involvement of civil society (including members of the opposition political party), and an emphasis on sectoral performance. But where they need further work is in developing links between, on the one hand, the activities and outputs of the health ministry and its agencies and, on the other, the overall performance of the 6 sector as measured by sectoral development indicators. Identification of such links would assist sectoral planning, resource allocation, line management, service delivery and accountability. The development of M&E capacities within GOG is so important because it will help such links to be identified. Step 5: Identify M&E activities of central and line ministries and other organizations As discussed briefly above, there appears to be a need to improve the functioning of sector ministries' PPMEDs. The skills mix of staff in these units is not known with any clarity. Both Bank missions heard comments on the difficulty in attracting and retaining good staff in such units, and this appears to be related in part to civil service salaries, career streams, and the low value placed by some sector ministries in having available M&E findings to support their internal decision-making thus low demand for M&E is not conducive to a high quality of supply. The converse also applies, of course high-quality supply would be wasted unless there is real demand for M&E findings. This is a chicken-and-egg issue, which is addressed further under step 7 below. As a senior official in the Finance Ministry noted, if the heads of ministries viewed M&E as important to their work, they would take more steps to ensure the function was adequately resourced. Key sources of data available to sector and central ministries include, for example: the data collected by PPMEDs and ministry statistics units; beneficiary surveys conducted by MDAs under the aegis of the Office of the Head of the Civil Service; the Ghana Living Standards Survey, which is conducted every four to five years by the Ghana Statistical Service (GSS); and the Core Welfare Indicators Questionnaire (CWIQ), conducted annually by the Ghana Statistical Service. The State Enterprises Commission (SEC) provides a model of performance measurement and oversight in Ghana. The SEC oversights the performance of state-owned enterprises, and the framework includes the following: clear objectives for each enterprise, together with a set of agreed performance indicators; preparation of corporate plans and of linked performance agreements at the organizational and individual levels (with performance bonuses for staff if targets are achieved); quarterly reports by each enterprise on its performance, on the basis of clear reporting guidelines; and annual assessments by SEC and (separately) by the relevant sector ministry of the performance of each enterprise. This model demonstrates the GOG's ability to set up a sophisticated performance management system; the actual performance of this system remains to be investigated. GOG plans to apply a broadly similar type of system to subvented agencies, as part of the public sector management reform program which the World Bank is actively supporting. 7 The Ghana Audit Service (part of the finance ministry) is working, with the support of the Swedish National Audit Office, to build its technical skills in the areas of financial and performance audits. This raises the prospect that at some future date the audit service could play a valuable role in assessing the M&E systems within individual MDAs. Such assessments would ideally encompass the management and adequacy of the M&E systems, and particularly the accuracy, reliability, objectivity and timeliness of the information which such systems provide. Once M&E information and findings become more important for GOG for purposes of sectoral planning and resource allocation (i.e., performance budgeting), line management and service delivery, and accountability there will be a heightened priority in ensuring that the information produced by MDAs is reliable.3 The Ghana Institute of Management and Public Administration (GIMPA) provides some M&E training to civil servants and NGO employees, and has the potential to provide a wider range of M&E courses, including trainer training in M&E. Organizations such as GIMPA thus have the potential to be important sources of supply of M&E specialists within Ghana. Favourable mention was made to both Bank missions of the high quality of a number of GIMPA staff. Mention was also made of the potential of GIMPA to assist in the provision of logframe training for officials, and this is planned as part of the MTEF training. GIMPA is also being funded under the Bank-supported PSMRP to provide a range of types of training for senior public and private sector executives via distance learning. Various other research institutes were also mentioned, particularly as a source of sectoral expertise. It is notable that the training provided to ministries in the lead-up to the introduction of the MTEF in 1998, on objectives, targets, outputs and indicators, was provided by GOG officials themselves. This entailed a collaborative approach involving staff of several of the key central agencies. Another round of training for senior officials, also provided under this collaborative approach, is currently underway. Supply issues and some supply options are considered further in step 8. 3Where line ministries use data they produce to support management decisions and for the internal allocation of resources, they will have an incentive to set up procedures (unless they already have them in place) to assure the accuracy, objectivity and reliability of the data this might well involve internal audit units. Where data are used externally for reporting to parliament and to civil society, and as inputs to GOG policy decisions including budget resource allocation (via the MTEF) there also exists a need for external quality assurance. Key actors here include the Ghana Audit Service, the budget areas of MOF, NDPC, the GSS and various parliamentary committees. Pressures are likely to grow for these bodies to strengthen their oversight roles and capabilities. 8 Step 6: Identify M&E activities and roles of development assistance agencies The extent to which donor requirements for M&E have helped build up GOG's own M&E capacities, including within PPMEDs, is not known in any detail. GTZ is funding the employment of a full-time international expert in M&E in the agriculture PPMED. CIDA is supporting the development of policy skills of selected central and line agencies, including their use of M&E findings in policy analysis. And the World Bank is supporting a pilot project which is, in part, intended to improve the M&E functions of several district assemblies and the district offices of central government ministries. However, a number of donor representatives and some officials have acknowledged that the multiple and diverse donor requirements for project M&E are a continuing burden for ministries. On the broader public sector reform side (discussed in more detail in the next step) donors are very active, and many of the reform initiatives which donors are actively supporting have a performance orientation. For example, DFID is working to support GOG's work on the MTEF, the civil service performance improvement programme (CSPIP), and the further development of statistical systems. UNDP is working actively on the national governance program, which includes a component to assist civil society, and the EU, GTZ, CIDA and various NGOs are working on the local government decentralization reform. The World Bank is also heavily involved in supporting the public sector management reform programme (PSMRP). Step 7: Identify major public sector reforms in recent years A unique feature of the environment in Ghana is the substantive and long list of public sector reforms initiated in the past several years. The reforms include, among others: · Ghana's MTEF, including portfolio-based performance budgeting, which will simplify inter-sectoral resource allocation prioritization and decision-making. It will also facilitate a sharper focus on sectoral objectives, priorities, promised performance via explicit performance (output) targets and on the actual performance which is delivered. The MTEF also has the potential particularly if accompanied by greater predictability in resource flows to support sound planning and management within sectors. In this environment, there is considerable potential for M&E findings to support more informed resource allocation decisions at the sector and project levels, and also to support a results-based approach to line management and service delivery · Ghana's PSMRP (supported by a World Bank loan), which seeks to change central government structures, responsibilities and objectives, and seeks to achieve a performance culture. It will actively pursue improvements in 9 service delivery, the setting of ministry performance targets, and results- based management within ministries · the Civil Service Performance Improvement Programme (CISPIP), which includes the formal identification by ministries, departments and agencies (MDAs) of modalities for improving their performance, with a strong emphasis on client orientation. It also includes performance agreements (between an individual and his/her Minister) at the level of heads of MDAs and heads of departments there are 31 heads of MDAs and 214 directors and heads of departments within GOG. The individual performance targets which are to be included in performance agreements are meant to be based on the MDA performance targets specified as part of the MTEF · the government has agreed in principle to a significant decentralization (i.e., devolution) of responsibilities to the regional and district levels. This will require the creation of a separate, local government civil service (eventually employing as many as 70% of all current civil servants) to support the work of the 110 district assemblies. One of the motivations behind this reform is to get government especially service delivery closer to the people. One of the quid pro quos for devolution is likely to be greater accountability for performance. It will be important for district assemblies and for government departments at the district level to have the capacity to assess public sector performance, although their reportedly weak overall capabilities are an important impediment to this. The World Bank is currently supporting a poverty-reduction project which is designed to build M&E capabilities in three pilot districts; this has the potential to build and to demonstrate good practice M&E at the district level · GOG's agreement to a CDF approach constitutes another type of public sector reform. The CDF approach has manifested itself in a closer dialogue involving donors and government the health sector appears to constitute a good practice model of how such dialogues might develop. The CDF principles include a focus on development effectiveness and a performance/results orientation, from the shared perspective of GOG, donors and civil society. The most recent development with the CDF in Ghana is a process of engagement with civil society including the parliament and NGOs4 the World Bank and other donors are providing active support for this process. Discussions indicate a strong desire within civil society groups to focus on the performance of government on sector outcomes and impact, and not narrowly on the outputs of government projects and on the achievement of development goals. This keen interest provides a valuable opportunity to expand the focus of efforts to build M&E capacities and orientation to expand them from a focus on GOG to also encompass civil society · finally, the CDF focus on development effectiveness results will be reinforced by the government's preparation of a Poverty Reduction Strategy Paper. This paper will map out the obstacles to poverty-reduction and the 4This process builds on the SAPRI dialogue initiated in Ghana in 1997. 10 proposed policy responses to them. It will identify intermediate and final outcome indicators, the role of civil society in monitoring these outcomes, and the capacity building and technical assistance necessary to support the strategy. Together, these reforms provide both an opportunity and a constraint on opportunities for building M&E capacities. They provide a unique opportunity because of their breadth of scope and their commonality in the area of performance measurement (i.e., M&E broadly defined) thus the common dimension of, and opportunity from, the efforts to better manage public sector performance is the need to better measure public sector performance. This common dimension of performance measurement provides an opportunity for synergy between the various reform elements: it is difficult to build a performance culture within any government, but where there exist related reforms which have this as one of their goals, the reforms can work together in a mutually-supportive manner. But the ambitious nature of the reforms is also a potential constraint for building M&E capacities. This is because of the danger of 'reform fatigue' within government, particularly from the perspective of individual civil servants who may feel overwhelmed by a continuing wave of reform efforts. There is no easy answer to this possibility, except to argue that any explicit emphasis on M&E capacity development should be validly presented as providing support and further underpinning to the existing suite of public sector reforms in Ghana, and does not constitute a new reform initiative. This view was put to senior officials during the missions their receptivity to this proposition is evident from the strong support from them for assistance in establishing an effective M&E framework in Ghana (discussed below under step 8). Another proposition which was put to senior officials and to donors during the missions and to which there is general agreement is that the public sector reforms collectively constitute an impressive architecture for public sector management. There is also agreement that the necessary next step is to deepen the reforms, to ensure the new systems and modalities (such as performance agreements) are of high quality, are more universally applied, and that they work as intended. Step 8: Demand for and Supply of M&E Demand The missions concluded that there is a strong level of demand for M&E within GOG at the senior official level within the key central agencies World Bank assistance in developing an effective national/sectoral M&E framework is sought by officials representing the Policy Management Group, NIRP secretariat, MOF and NDPC. The level of support which might exist from the heads of sector ministries and of departments and agencies is not known. 11 An important next step is to secure the formal agreement of a powerful central agency to act as the 'champion' for M&E this will enable GOG to move forward decisively with M&E capacity building. It is important to have an M&E champion, to ensure a continuing focus and priority on performance measurement as part of sound governance. Part of being a champion involves being a cheerleader vis-à- vis the rest of the public sector. It also involves having a powerful agency or ministry being responsible for ensuring via encouragement and/or force that the rest of the public sector moves from rhetoric to action. A champion must be able to identify good-practice approaches to M&E and to disseminate them widely and convincingly to sector ministries. Another important source of demand for M&E is civil society. At the workshop which the Bank hosted for civil society during the October mission, clear requests were made for donor support in helping NGOs build their capacities. This included basic capacities such as communication, fundraising, policy advocacy and the building of alliances. They also included the strengthening of their capacities to assess public sector performance, via M&E and budget analysis. (Some powerful support for such capacity-building was also expressed by senior officials at the subsequent workshop for them.) These requests are consistent with the emphasis on the `voice' of civil society emphasized in the CDF. An important plea was also made at the civil society workshop: NGOs, think- tanks and parliamentarians requested the active support of donors in encouraging GOG, ministries and civil servants to allow a greater openness and accessibility to information on government activities and performance. Considerable concern was expressed about a culture of secrecy in ministries, and about the lack of a service orientation. Supply The supply of M&E skills and information is unclear. GOG does appear to have a number of staff with an understanding of the importance of setting organizational objectives and of measuring performance, and the fact that it has been able to organize some training in these areas is an encouraging sign. PPMEDs contain a number of staff with statistical skills and experience in collecting performance indicators. Higher-level skills in the areas of policy review and formal evaluation would seem to be much scarcer certainly, GOG itself has identified the shortage of policy analysis skills as a key capacity issue to be addressed. CIDA is supporting the strengthening of policy analysis skills in a number of agencies, including the PMG, Cabinet Secretariat, MOF, NDPC and selected PPMEDs. As noted above, GIMPA provides M&E training to civil servants and to NGO employees, and has the potential to expand the range and types of M&E training which it provides it has the potential to become an evaluation centre of excellence, both for Ghana and for West Africa. Other research institutions and 12 think-tanks, including NGOs such as ISODEC, NDI and IEA, have a growing capacity to undertake budget analysis work. The above steps lead to a number of options for the development of M&E capacities. 13 OPTIONS FOR DEVELOPING M&E CAPACITIES A particular issue to bear in mind is that M&E capacity development is a long- term proposition it is not something achieved with a short-term technical assistance project, but rather will require a sustained, ongoing commitment by GOG and whichever donors are able to actively support this work. Another consideration is that the simultaneous pursuit of a number of M&E capacity development options would be valuable in creating the critical mass necessary for sustainable change. A number of options are outlined below; they overlap to some extent: 1. GOG policy statement on M&E/performance measurement Such a statement could be relatively easy to obtain, particularly if it came from the President or Vice-President, strong supporters of the reforms. A policy statement would require active dissemination, via such means as a budget circular, guidance on CSPIP and performance agreements, etc. It might also be possible to link such a statement to the imminent civil service code of conduct and ethics. Pro: A policy statement would provide a high profile to M&E and reinforce its importance to government. It would also help to provide a clearer understanding to civil servants of the links and synergies between various key reform programs. It would also provide a vehicle for GOG to articulate its policy on the extent to which information on government activities and performance is to be accessible to civil society. Con: There may already be a surfeit of policy pronouncements, and it might engender cynicism to have one more, on M&E. On balance: A policy statement is unlikely to do harm. It might provide some clarification to civil servants trying to 'connect the dots' among the various public sector reforms. It might also provide some support to those working in M&E functions (such as in PPMEDs), who can often feel isolated and under-valued. It would provide further support to the `voice' of civil society. 2. Development of an effective M&E framework within GOG An M&E framework might desirably set out clear responsibilities for different actors within GOG who is expected to do what, why, when, and how. It could encompass the planning, conduct, quality, reporting and uses of M&E findings, at the national, sectoral and regional/district levels. It could clarify basic or common data needs, including the respective roles of central agencies, sector ministries including PPMEDs, other MDAs, district assemblies, the parliament and civil 14 society.5 Development of an M&E framework could also be a vehicle to achieve a simplification of donor requirements for M&E. The NDPC has sketched an outline of an M&E framework contained in Vision 2020 although there is little evidence that it is effective. This experience indicates the need for caution in attempting to develop a more effective framework. Pro: A framework which was adequately policed by a powerful champion could have real 'bite' to move from rhetoric to action. It would help to elevate M&E to be an explicitly important function of government. Con: For such a framework to function well, it would require a high level of support among MDAs, particularly among central agencies, and would also require the commitment of very scarce GOG resources. Such support and commitment has not been evident in the past. On balance: Before a substantive M&E framework and infrastructure is established it will be invaluable to build up demand within GOG and to establish some good-practice pilot approaches within ministries. The PPMEDs provide the basis for part of that infrastructure, although there is no clear evidence that any of them can yet be regarded as constituting a good practice approach. The establishment of a comprehensive M&E framework should probably be regarded as, at best, a medium-term objective. 3. Identification of M&E champion(s) within GOG A champion could be a key central agency, secretariat or committee, as suggested above, or a powerful individual such as a Minister. A champion would need to have a genuine interest in, and commitment to, the creation of a performance culture via an emphasis on M&E. For a champion to play an effective role there would need to be clear expectations about what were its objectives and functions these might usefully include the advocacy of the need for a performance orientation in GOG reforms, oversight of the progress of MDAs with conducting and using M&E findings, a mandate to 'win hearts and minds' among senior officials, encouragement/ support for the community of M&E practitioners within GOG (especially within PPMEDs and also in civil society), provision of advice to GOG on further development of M&E within government, and the initiation of major sector reviews/evaluations according to emerging policy priorities. It would also be desirable that a champion promote the `voice' of civil society. Experience in other countries is that an effective champion organization need not require many additional resources to perform its functions effectively but 5 Future or subsequent tasks might include preparation and dissemination of support/guidance material, including standards for M&E. Related to this is ensuring the accuracy, reliability, objectivity and timeliness of, and some consistency in, the M&E information provided by different MDAs. 15 equally, unless M&E is assigned some clear profile and importance a champion would be unlikely to achieve much. Pro: Existence of a pro-active champion would ensure that M&E remains on the reform agenda, that opportunities to promote a performance culture within the government are exploited to the fullest possible extent, and that M&E makes a substantive contribution to achievement of the reforms. Con: There could be presentational problems if it were perceived that a champion had failed to deliver significant progress with the conduct and use of M&E within GOG this is an argument for realistic expectations from any champion. On balance: Nomination of a key central agency, secretariat or committee as the M&E champion is necessary to obtain and maintain a high profile for M&E, and to embed it within government. 4. Deepening of the performance orientation within the MTEF The MTEF is likely to play a significant role in fostering and mandating a strong performance orientation in the annual budget process in particular via its emphasis on objectives and outputs. While the MTEF architecture is sound, senior MOF officials (including a senior official of PUFMARP) and the head of the NIRP secretariat agree that there is a need to deepen the MTEF's emphasis on performance the latter asked for World Bank assistance to ensure that GOG can start to report its sectoral ministry performance by the middle of 2000. A stronger MTEF focus on performance is currently being pursued via logframe training for senior officials, focusing on the objectives of, and desired outputs from, government activities. The training is being provided by officials from several central agencies, and is supported by DFID among others. However, the quality and depth of the training is uncertain, and there have been some reports that participants have not reacted favorably to it. There could be merit in provision of logframe training on a more systematic and widespread basis to senior and middle-level officials, including managers and policy analysts. The training would also benefit from an explicit focus on the range of different M&E tools which exist, and on their relative strengths and weaknesses. Thus there would be merit in the content of the MTEF training being deepened, and its availability broadened. There might also be merit in including some representatives of civil society as participants in the MTEF training. This could help ensure that a user and service delivery perspective is included in the logframe analysis of government activities. It could also help to increase the level of understanding which civil society representatives and government officials have of each other's perspectives and priorities, and of the realities of, and constraints on, government service delivery. 16 MTEF training is part of the answer but it is unlikely to be sufficient. There are issues of the adequacy of the flows of M&E information and findings from sector ministries to central agencies such as MOF and NDPC, and to groups such as PMG and COSCO. There is also the issue of the extent to which MOF and other central agency staff make use of such information. These issues in turn raise questions about the nature of the frameworks and requirements for sector ministries to provide M&E information to central agencies, and about the extent to which central agencies and others are able to make good use of such information there is little point in mandating information flows if the information is not used effectively. These important topics will require close attention at some time in the future they essentially relate to the need for an effective M&E framework (option 2 above). For the meantime, provision of deeper and more MTEF training should help in raising awareness of M&E and what it has to offer to policy-advising central agencies and to sector ministries. This in turn should help to further build the demand for M&E. Pro: The MTEF has the potential to influence the manner in which resource allocation decisions are taken the budget system represents a key forum in which M&E findings can make a substantive contribution. For this potential to be realized it is necessary that those who use the MTEF and input to it have a solid grasp of performance measurement concepts, uses and limitations in underpinning sound policy analysis and advice. A particular issue which training could address is the desirability of moving beyond a focus on outputs to also addressing outcomes and impact. More widespread training would also have the potential benefit of fostering a clearer understanding within MDAs of their own objectives and performance yardsticks. Con: The precise extent/scope/quality of the MTEF training currently being provided needs further investigation before specific options for additional training can be addressed. The role of DFID and the potential value-added of World Bank support would also need to be mapped out clearly, in conjunction with the PUFMARP steering committee. On balance: The MTEF is a significant and high-priority initiative currently at an important stage of development and implementation. The training now being provided should be examined more closely to ensure it is progressing far enough and fast enough to support the successful implementation of performance budgeting under the MTEF. 5. Review the structure, functions, resources and practices of PPMEDs Sector ministries appear to differ significantly in the priority they attach to their PPMEDs some ministries have not even established one and to their size and functions. PPMEDs also appear not to collect data on service delivery by parastatal organizations thus ministry data present an incomplete picture of sector performance, especially for sectors such as health where parastatals play a significant role. 17 As noted earlier, the lack of specific new initiatives to support PPMED functions appears to be a gap in GOG's public sector reforms. A review of PPMEDs has the potential to highlight good-practice approaches to the planning, conduct, quality-assurance and use of M&E findings. Such a review would encompass, among other dimensions, a review of statistical systems including methods of data-collection, processing, storage and reporting various donors (including the World Bank and DFID) also have a strong interest in supporting such capacity-building. Reviews of statistical systems would seek to identify main uses and users of data, and would also seek to simplify the apparent multiplicity of indicators and data collection methods. A review of PPMEDs would probably need to address their role vis-à-vis the role of the Ghana Statistical Service. Suggested generic terms of reference for PPMED reviews are at Annex B. The 1998 GOG-donor health sector review identified the need to review and revise the approach to information management and to sector indicators the latter to focus not just on process-based indicators but also on output and outcome indicators. The need to measure the contribution of private and not-for- profit health services was also identified. A review of PPMEDs could either take an across-the-board approach of reviewing the functions and modus operandi of all PPMEDs, or alternatively it could focus on only a handful, selected either because they represent aspects of good/best practice, or because they have an immediate need to improve their performance. There would be a number of synergies between this option and option 6 (`Ministry mentoring'). Pro: A review could help identify resource implications and spread good- practice approaches more widely within GOG. It would lead to a clearer understanding and consensus of the value (or lack of value) provided by PPMEDs, and would also be a useful input to the future development of an M&E framework in GOG. In addition, if there is substance to the suspicion that much of the data-collection work of some PPMEDs is unproductive, a review would have the potential to provide substantial net savings to the ministries concerned. Con: A review would be one more burden on ministries. On balance: A review need not be a resource-intensive activity, and could yield net savings to ministries. It would foster a closer focus on M&E on the potential benefits from M&E, and also on the extent to which this potential has or has not been realized. A review could be conducted either for all PPMEDs or on a voluntary, pilot basis this would also identify the potential size of net savings achievable by other ministries. 18 6. `Ministry mentoring' Sector ministries play a key role in the management of the sectors they oversight. The health sector and the health ministry, for example, are frequently mentioned as examples of good practice because of the pooled approach to a portion of donor funding and the annual GOG-donor sector reviews with their central focus on sector performance. A `ministry mentoring' or twinning approach could entail a formal arrangement whereby, for example, a donor or number of donors agree to provide close support to a sector ministry in developing its M&E, review and sector policy analysis capabilities. One intention would be to enable a sector ministry and/or its PPMED to build its capabilities in a sustainable manner. A second intention would be to ensure that the ministry/PPMED represents good practice and provides a demonstration model to other ministries. Such demonstrations could become particularly important as the CDF, which requires more of a national/sectoral perspective, develops. GOG line ministries appear to operate very independently from each other, and so might be unlikely to be pro-active in sharing good practice lessons among themselves. Thus the effective dissemination of good-practice lessons is likely to require the active support and pushing of an M&E champion. The type of approach envisaged here would differ from a short-term project approach to capacity building (where, for example, an M&E unit is set up for the life of a fixed-term project), and would also differ from more informal support arrangements which may currently exist: · the formal arrangement would be between organizations the ministry and/or PPMED within GOG on the one hand, and one or more donors on the other. Within the World Bank, for example, the arrangement might include sector or evaluation experts within the Africa regional unit, the Bank sector board, OED, WBI and/or OCS · the arrangement would be expected ab initio to be long-term in nature, and not dependent on the continued availability of individual donor or GOG staff. Support could involve some or all of the following, depending on agreed sector needs and priorities: · close collaboration and assistance on sector reviews this type of 'action learning' approach would be expected to be more effective than more conventional training. It would also promote a greater performance orientation in MTEF (i.e. budget) examination of sector allocations. (In addition to the regular GOG-donor health sector reviews, the World Bank is also participating with GOG and a number of other donors in a joint review of the transport sector) · identification/sharing of good practice sector analyses, reviews, and evaluations, including from other countries 19 · joint GOG/donor evaluations of major projects another action learning approach · donor assistance in identifying and funding a consultant who would be available to provide advice/assistance/coaching on a part-time basis, and under a long-term arrangement. A consultant could be a Ghanaian national (desirably) or an expatriate. The intent would not be for the consultant to become a de facto staff member of the ministry/PPMED that would be more likely to establish a dependency relationship rather than build GOG capacity. A long-term relationship would help ensure the consultant becomes thoroughly fluent with the policy, management, staffing and technical issues relating to the ministry/PPMED, and is able to establish sound and ongoing working relationships with GOG officials · donor assistance with training for ministry/PPMED staff in M&E, policy analysis and sector issues.6 Assistance might include donor financial support for the cost of training in-country, at regional institutions such as the AfrDB or JAI, and/or via distance learning. Training could be provided at one or more of three levels: awareness-raising for senior/mid-level managers; training for implementation teams and advisory staff; and trainer training · support for study tours, to learn from good-practice approaches in other countries. If the support for 'mentored' line ministries were to be effective, it would be necessary that the PPMED for those ministries were reviewed as per option 5 it would be difficult to build the M&E capacities of a ministry unless its PPMED were operating efficiently and effectively. Three line ministries which might be good candidates for support could be the agriculture ministry (ministry of food and agriculture MOFA), the ministry of roads and transport (MRT), and the health ministry, which is already benefiting from close donor collaboration. Donor representatives considered that MOFA's capabilities are generally weak. One early opportunity for Bank support for MOFA is the agricultural sector investment program currently in preparation. Another is the apparent need for a joint GOG/donor review of the agriculture sector (similar to the health sector reviews), as reported by the Bank's country director. A special feature of MOFA which is of interest and which might yield lessons for other ministries is the apparently substantial emphasis which it devotes to data collection at the district level. This is relevant to GOG's strong emphasis on decentralization and participation. Note that FAO has a leading role in helping MOFA build its statistical capabilities the scope of this work would need to be investigated further. Finally, GTZ is funding the employment of a full-time 6A list of specific areas of M&E in which training could be provided is summarized under Option 7 and is discussed in more detail in Annex B. 20 international expert in M&E in MOFA's PPMED. Thus the need for additional donor assistance to MOFA would have to be investigated carefully. The roads and transport ministry is another possible candidate for support. A number of donors have already expressed interest in a more collaborative approach in their work with this ministry, and there is potential for it to learn from the types of approach adopted with the health ministry. The health ministry overall has a reputation among donor representatives for a relatively high level of capability it therefore has a relatively high starting point on which further support could build although its PPMED is regarded by some donor representatives as weak, and ministry officials have suggested there are important weaknesses in its statistical systems. GOG's health sector reform program includes the objective of strengthening M&E capacities. Support for line ministries would also be expected to ensure a high standard of input from them in meeting MTEF requirements for performance measurement and in framing and meeting ministry performance improvement plans and individual performance agreements (as part of CSPIP). There would also be merit in providing some support to central agencies such as MOF and NDPC they are important players within GOG, and in one sense are too important to risk being left behind if a strong performance orientation and culture is to be developed in the civil service as a whole. There appears to be a need and an opportunity to build a performance orientation within MOF and to improve its policy analysis capabilities. This would enable it to play a substantive and more frequent role in sector reviews. It would also stand MOF in good stead in the medium-term, when it presumably will have responsibility for the oversight and management of the MTEF. The difficulties experienced by NDPC might be assuaged if its staff were given the opportunity to enhance their skills via participation in sector evaluations and reviews. Donor support could enable NDPC to rely less heavily on sector ministry inputs to national assessments. Pro: Would provide intensive support to selected GOG ministries/PPMEDs, with a greater probability of achieving a sustained increase in ministry capabilities. Con: The intensive approach to donor support would be costly to donors. On balance: Such intensive support would appear to offer good prospects for sustainable capacity-building within GOG. Provision of intensive donor support to selected line ministries would probably be much more effective if it was undertaken in conjunction with a review of their PPMEDs. 21 7. Skills enhancement via provision of M&E training During both missions officials from a range of ministries emphasized the need for skills training. Representatives of civil society including not only NGOs but also research institutes, think-tanks and parliamentarians also expressed the hope that they would be given the opportunity to participate in training courses. The requests for M&E training could be met either via donor (and GOG) direct provision, or via donor (and GOG) support for establishment of a training fund. Provision of a substantial amount of training would appear to be an indispensable element of efforts to strengthen Ghanaian M&E capacities. Issues which need to be addressed include: the types and depth of M&E training to be provided; the number of participants to be trained, and the ministries and non-government organizations to be targeted for priority access to the training; and arrangements concerning how the training would be managed. An important part of the management of any training provided would be continuing evaluations of the appropriateness, targeted audience, quality and depth of the training provided at the very least, to ensure actual learning has occurred, and also to ensure use on-the-job of skills acquired. A second important aspect of this work will be the strengthening of indigenous capacities to provide this training i.e., the transfer to Ghanaians of the capacity to provide this training, via trainer training, wherever current skill levels are not yet sufficiently strong. Trainer training would best be targeted towards selected partner institutions. It would be possible to set up a training fund to which donors could contribute and which could be jointly managed by GOG and donors, or managed solely by GOG. Funds could be made available to ministries/PPMEDs which are able to establish a high priority/need for support in a competitive process, against specified criteria, and subject to GOG's own sector/ministry priorities this approach could help avoid diffusion of the impact of the training provided. For ministries/PPMEDs which are given initial access to training opportunities, further access would desirably depend on demonstrated performance in acquiring and utilizing skills. Decisions would need to be made about: · the types of M&E training for which support would be provided · the depth of training and the mix of officials and others for whom training would be provided such as awareness-raising for senior/mid-level managers; training for implementation teams and advisory staff; trainer training; training for NGOs, think-tanks and others in civil society; and awareness-raising for ministers, parliamentarians and journalists · the types of Ghanaian training provider which would be eligible for support. The types of training provided could include: 22 · logframe analysis of organizational/program/project objectives, intended beneficiaries, results chains, assumptions and risks7 · performance indicators · statistics/data collection surveys, sampling, questionnaire design · rapid review techniques · participatory/stakeholder evaluation · sector policy analysis and review · formal approaches to evaluation cost-benefit analysis, program evaluation. Pro: Provision of a substantial amount of training appears to be indispensable. Either a broad-based or a more tightly targeted approach to skill-building could be adopted. Con: The extent of support necessary to strengthen the skills base within GOG is unclear. Also, GOG has experienced difficulty in retaining skilled staff;8 it is possible that those officials who receive intensive M&E training would be tempted to leave GOG for higher salaries in the private sector. Staff losses would create a short-term problem for ministries/PPMEDs but might not be regarded as a problem in the medium-term if GOG is prepared to contract M&E functions from the Ghanaian private sector, universities and think-tanks. On balance: A substantial volume of training appears to be indispensable. A training fund has the potential to support targeted capacity-building, once reviews of the roles and functions of PPMEDs have been completed. 8. Support establishment of a Ghanaian evaluation forum/network The missions gained the impression that the staff of PPMEDs feel isolated from their counterparts in other ministries. Experience from other countries is that there are benefits from encouraging interaction among practitioners and managers of M&E from both inside and outside government and that interaction can be fostered through the creation of an evaluation forum or network. In Australia, an evaluation forum met monthly in the national capital, in two-hour meetings during working hours. At each meeting several speakers would make short presentations on topical issues of broad interest, together with a question & answer session and announcements of recent publications/studies. The large majority of forum participants were government officials, though some private consultants also attended. The forum was organized by a voluntary committee composed of representatives from a range of government departments. The direct and indirect costs of the forum were modest. 7A closely related variant of this approach is `results-based management'. 8GOG intends to address the retention issue via action such as higher salaries for civil servants; in the short to medium term, however, the issue of retention and skill development will be complicated by the substantial downsizing planned for the civil service and the broader public sector. 23 A Ghanaian forum might eventually encompass the creation of a website and even a journal. Such a forum or network could have a number of advantages: · to help reduce barriers (vertical stovepipes) between ministries and PPMEDs · to help establish an M&E community, sharing insights into issues and problems commonly encountered, including good practice approaches to dealing with them · to facilitate understanding of the role of M&E within GOG, including its importance to sound governance and its links to other public sector reforms · to increase civil society's understanding of the operations and performance of the public sector · to foster public/private sector interaction · to help create greater awareness of the demand for and supply of persons with M&E skills to support a better functioning M&E labor market. A Ghanaian forum could associate itself with the mooted African Evaluation Association. A pan-African association was proposed by a number of participants, including representatives of the Kenyan Evaluation Association, at an African evaluation conference in Nairobi in October 1999. A number of Ghanaians participated in the Nairobi conference, with some support from the World Bank, and they have expressed their intent to set up a Ghanaian evaluation association, with a secretariat provided by ISODEC, a leading NGO. A national evaluation association could either be a complement to, or even a substitute for, the forum proposed in this paper. Pro: An inexpensive means of developing an M&E community within GOG and more broadly within Ghana. Con: One more activity for busy civil servants. On balance: A useful idea, which might require, at most, modest 'seed money' for a limited period. The M&E 'champion' could play a role in the support of a forum. 9. Creation of an evaluation foundation An evaluation foundation could be set up to identify and commission selected strategic evaluations i.e., major evaluations of a small number of strategic GOG programs or activities. The steering committee for the foundation might include some or all of: GOG representatives, including senior officials, parliamentarians, representatives of think-tanks, NGOs and the private sector. There are at least two possible approaches to the creation of an evaluation foundation. The first is to set it up as a very high-level body which ensures that the evaluations and reviews which are important to GOG are conducted, and that these evaluations ask the right questions. A second approach would be to set up a foundation independent of government as an external source of review of 24 public sector performance; the foundation would commission reviews by think- tanks, universities and NGOs. Under the first approach, a foundation would provide a vehicle to the government to ensure that evaluations and reviews of key, high-priority and high-profile government activities are carried out, that they are conducted to a high quality, that they focus on the main issues important to government, and that they are completed in a timely manner. Countries such as Australia, Canada and the United Kingdom have relied on sector ministries to conduct most of the evaluations and reviews of sector activities though often with some central agency participation in the management or even conduct of the evaluations. But governments of these countries have also commissioned largely independent reviews, with some sector ministry participation, of strategically significant programs and activities, including those with important cross-sectoral impacts. As noted earlier, a concern of GOG is that a number of Cabinet decisions have not been implemented, or not implemented effectively. This partial frustration of GOG's policy priorities has led to the creation of the PMG and of COSCO, and it might be expected that either or both of these groups could play a leading role in the management of an evaluation foundation under this approach. The second approach to creation of an evaluation foundation would provide an opportunity to engage civil society in assessing public sector performance at the highest levels. This would strengthen the opportunities for the views of civil society and program beneficiaries to be heard by GOG decision-makers and program managers. It might also foster a keener appreciation among civil society representatives of the opportunities for, and constraints on, more effective performance in the public sector. Although civil society organizations could decide on their own to undertake such evaluations and reviews and several NGOs and think-tanks are already taking steps to improve their capacities to undertake analyses of the annual GOG budget such work would be much more effective and could contribute much more to GOG if it had the active collaboration and support of GOG. Pro: Creation of an evaluation foundation is a means to ensure that independent evaluations and reviews of key GOG activities and programs are conducted. A foundation could be set up to ensure either that the evaluation priorities of GOG are met, or that the priorities of civil society are met, or both. Con: The level of support within GOG for inclusion of civil society in evaluating public sector performance is not known. Sector ministries might be expected to resist external evaluations and reviews of sector programs, irrespective of whether such evaluations were commissioned by a GOG-led foundation or a foundation led by civil society. 25 On balance: An evaluation foundation would provide a powerful vehicle to commission high-quality and timely evaluations and reviews of key GOG programs. The extent of involvement of civil society would require careful consideration. 10. Support for civil society involvement in performance measurement Civil society, broadly defined, includes stakeholders such as ordinary citizens, NGOs, the media and parliament. Citizens are the beneficiaries of government activities, either through direct consumption of government-provided goods and services, or more indirectly through government regulation and other activities. Citizen feedback is invaluable in identifying which government programs and activities are working well, which are not, and why. The importance placed on the views of citizens is illustrated by the beneficiary assessments being conducted as part of the CSPIP initiative. Parliament, NGOs and the media also have an enduring interest in government performance. The ongoing CDF discussions which have built on the SAPRI (Structural Adjustment Participatory Review Initiative) dialogue initiated in Ghana in 1997 provide an additional opportunity to involve civil society in the assessment of government performance, and more widely in discussions about national and sectoral priorities and progress in achieving development goals. Such interaction between civil society and government (and donors) means that civil society is also an important potential user of information on the performance of government i.e., of performance indicators and the findings of evaluations and reviews. The involvement of civil society would help to put pressure on GOG and on civil servants to better manage inefficient or ineffective government activities and thus achieve higher levels of performance. Civil society can also be instrumental in putting pressure on any government agencies which are suffering from corruption. During the Bank's mission to Ghana in October 1999 a workshop was conducted for representatives of NGOs, think-tanks, parliamentarians and others from civil society. The workshop examined good-practice examples from Ghana and other developing countries of civil society involvement in assessing public sector performance. The key part of the workshop was the brainstorming group sessions, in which participants were asked to identify the role that civil society wants to play in assessing public sector performance, to identify the main impediments to playing this role, and to outline priority steps for achieving deeper civil society engagement in this area.9 A key impediment identified by workshop participants is the limited capacities of NGOs and think-tanks in even basic activities such as communication, policy advocacy, fundraising and networking. Participants made it clear that these 9The World Bank will soon publish the proceedings of this workshop, including the resource material on good-practice examples. 26 areas would need to be addressed if NGOs and think tanks are to be able to contribute more fully to the assessment of public sector performance via more sophisticated tools such as M&E. Donor support was requested to help build basic skills as well as familiarity and expertise in M&E. One issue for civil society and donors to consider is the degree of familiarity and understanding of M&E concepts and tools which these groups need in order to make full use of the M&E information available to them. It is necessary for users of such information to have at least some understanding of its strengths, limitations and weaknesses. Civil society's involvement with such information can reasonably be expected to be more than as users or observers of information provided by others. These stakeholders have a role to play in inputting their views and opinions into evaluations and reviews of government activities particularly in the case of ordinary citizens as consumers of government goods and services. As suggested under option 9 above, civil society also has a role to play in influencing the evaluation agenda identifying particular government activities whose performance needs to be evaluated or assessed. The World Bank and other donors are accumulating a growing body of experience in interacting with civil society, particularly at the grassroots level, but also increasingly at national and regional (i.e., sub-national) levels. In Uganda, for example, NGOs have provided inputs into the country assistance strategy and have helped identify steps to address the leakage of government funding for social services. A key element in encouraging such participation is the extent of government support for civil society to become more fully engaged in debates of government performance and policy priorities. The World Bank, with the active support of CIDA, the OECF and other donors, is supporting GOG in engaging closely with civil society in elaborating Ghana's country development strategy.10 This process of engagement will proceed in all 10 regions of Ghana. Assistance from donors and GOG to encourage civil society's involvement in policy dialogue and in oversighting government performance could be provided in a number of ways. These include, for example: · support for the creation of structures/frameworks which encourage the participation of NGOs in policy dialogue and advocacy · provision of M&E awareness-raising training and other NGO capacity-building support · greater involvement of civil society in sector reviews (as per the health sector) · support for the publication and dissemination of M&E information on government performance 10This is also integral to the CDF and PRSP processes. 27 · a public awareness and education campaign · support for NGOs, think-tanks and universities to build specialist M&E skills as well as expertise in budget analysis and other types of policy analysis. The parliament and media can also play an important role in these dialogues, and various types of support are possible. These might include: · awareness-raising activities, on issues such as the concept of performance, the uses of performance measurement (i.e., for management and service delivery, for sectoral planning and resource allocation, and for accountability purposes) · capacity-building support for the parliamentary service in a range of M&E techniques and in budget and other policy analysis. In addition, NGOs could work directly with parliament in helping it understand and digest available information on the performance of MDAs. South Africa provides an example of this type of interaction a think-tank there, IDASA, undertakes and publishes analyses of the South African budget. It also acts as a regional centre to encourage think-tanks in countries such as Ghana perform similar functions, and it has recently set up a network of African think-tanks to support budget analysis work. Pro: Civil society has a legitimate role in debates about public sector performance, and in helping to identify ways of improving performance. Civil society has a key stake in M&E findings and information, and could make an important contribution to the development of a performance culture within government. Con: The extent of civil society engagement in assessments of public sector performance is often a sensitive issue, particularly in countries where public access to information about government is neither automatic nor easy. On balance: GOG has committed itself to an intensive dialogue with civil society in elaborating Ghana's country development strategy. There exist many opportunities for closer engagement of civil society of which closer involvement in M&E is one. Such engagement can play an invaluable role in fostering a performance culture within GOG. 28 CONCLUSIONS The development of M&E capacities is an important step in supporting and deepening the performance orientation of GOG's public sector reforms. The ten options identified above are not presented as an overall action plan for Ghana, although they could be pursued in that manner if GOG chose to do so. There is a trade-off between the synergies from pursuing these options simultaneously and the burden of tackling them in that manner. There are issues of what options are related and can be clustered, and what would be a desirable sequencing. One question, to which there is no simple answer, is what is the minimum amount of M&E capacity-building efforts if sustainable increases in M&E capacities are to be achieved? It might be premature to attempt to design a new M&E framework (option 2), given that there is a lack of knowledge about what good practice M&E approaches within GOG might look like this will require further experience with alternative approaches before it is clearer what works well and what does not, and why. It is suggested here that a useful strategy to developing good practice examples which would provide a demonstration effect throughout GOG would be to target M&E capacity development efforts to a small number of ministries/agencies where there are good prospects for early success. This would be a particularly powerful approach where the senior management of the ministries are strong believers in the benefits of M&E. This could be pursued by the combination of several options as the first phase of a longer-term approach: · identification of a capable M&E champion at the central government level a central ministry or high-level committee (option 3). This is an important next step for GOG if it wants to move forward decisively with M&E capacity building · deepening of the performance orientation within the MTEF (option 4) · `ministry mentoring' (option 6), entailing the selection of several ministries for targeted assistance. The nature of this assistance would depend on the strengths, weaknesses and opportunities facing each ministry, as well as on the depth of support which donors are able to provide · a review of PPMEDs within the ministries chosen (option 5) · provision of M&E training (option 7). This combination would benefit from the existence of a GOG policy statement on the merits of M&E/performance measurement (option 1). Such a statement would help to legitimize the development of M&E capacities as a priority, and would signal the importance placed by GOG on the value of performance measurement and a performance culture. A statement and the existence of a powerful champion would also help to ensure that the first phase was viewed as exactly that as a first step potentially (and desirably) leading eventually to a 29 more comprehensive approach throughout GOG. A more comprehensive approach might be feasible within, say, five years. This combination of options could also benefit substantially from the development of civil society's capabilities to make full use of M&E information and tools and to input fully to debates of development priorities and of public sector performance (option 10). Civil society could also play an important role in an evaluation foundation which commissioned evaluations and reviews of public sector performance (option 9). There would be considerable merit in pursuing these options irrespective of decisions on the other options identified. The CDF dialogue now underway provides an excellent forum for a deeper engagement of civil society. A cut-down version of the suggested combination of options for developing M&E would also be sustainable. This could encompass selected 'champion' line ministries, together with options 1 (policy statement) and 7 (training) in the short- term, followed soon after by options 4 (MTEF), 5 (PPMEDs) and 10 (civil society). The workshop which the World Bank hosted for senior officials in October 1999 obtained the views of a number of senior officials on these options, and this has led to the action plan proposed below. This action plan will now be the subject of discussion with GOG. At the same time, the extent to which donors are able to provide substantive support to the various options is also being investigated. 30 ANNEX A AN ACTION PLAN FOR THE DEVELOPMENT OF GHANAIAN M&E CAPACITIES Capacity- Output Impact Input Timing Unit Total building Activity Cost Cost (1) Support to an Preparation of GOG To reinforce the importance to National adviser 2000 $30,000 $30,000 M&E champion a ministerial policy GOG of M&E, and to clarify 2nd secretariat within statement on extent of availability of GOG quarter Policy Management M&E/measurement of info and M&E findings to civil Group (Office of the performance society President) for purposes of Oversight reviews of See Activity #2 below 2000 advocacy, oversight, selected ministries' coordination and policy, planning, M&E support departments (PPMEDs) Monitor the policy, Identify and share good Ongoing planning and M&E practice approaches to the activities of MDAs; planning, conduct, quality, use identify and and reporting of M&E findings encourage good practice Develop a national Clarify responsibilities of key 2005 M&E framework for actors within GOG who is GOG expected to do what, why, when and how (i.e., the planning, conduct, quality, use and reporting of M&E findings) 31 Capacity- Output Impact Input Timing Unit Total building Activity Cost Cost (2) Review the Reviews of PPMEDs · Improvement in ministry 4 international 2000 $650 per $117,000 functions, structure, in MOFA, MOE, MRT and PPMED policy experts (4x30d) day plus (int'l) resources and and MOF to focus on analysis, policy advice and expenses practices of selected priority uses + users line management ministries' PPMEDs of M&E findings, 4 national $200 per + $24,000 (draft ToRs are at including related stats · Higher-quality GOG budget consultants day (national) Annex B) data and systems decisions on competing (4x30d) (suggested ToRs budget priorities attached) (3) Ministry International/ national Assisted ministries better able 4 international/ mid- up to up to mentoring 3 sector experts hired on long- to plan and conduct their M&E national experts 2000 $650 $131,000 ministries jointly term, part-year basis functions; also better able to x45d per annum start, for plus per selected with the to provide advice, assess sector development for 3 years at least expenses annum for champion, plus MOF/ assistance, coaching effectiveness 3 years 3 years NDPC and some training on M&E including sector reviews Support from Support to be 2000 2nd Evaluation Offices of negotiated; could quarter donors able to `twin' incl assistance in with individual sector sector reviews, ministries sharing of good practices, joint project evalns, study tours, some training, secondments (4) Support for M&E Provision of M&E Increased knowledge and Range of M&E 2000 1st Indicative $150,000 training training courses for competencies of persons who training courses quarter costings (indicative officials, NGO have received training provided in pilot to be only) staffers and stage; further dev'd for parliamentarians courses provided (i) WB 32 Capacity- Output Impact Input Timing Unit Total building Activity Cost Cost (draft training plan for on basis of needs and other pilot phase is at and cost- donors, Annex C) effectiveness (ii) GOG Arrangements for the ongoing planning, management and delivery of M&E training will need to be agreed Creation of an M&E training fund Development of 5- year training plan WBI evaluations of Identify cost-effectiveness and M&E training appropriateness of M&E training Institutional capacity- Support for a Strengthening of M&E Provision of 2000 1st building to support Ghanaian Evaluation capacities of a Ghanaian training materials quarter M&E training Center of Excellence institution able to provide and of trainer such as GIMPA training and other M&E support training; to provide M&E preparation of a training (via a `preferred medium-term supplier' `preferred supplier' agreement agreement) 33 Capacity- Output Impact Input Timing Unit Total building Activity Cost Cost (5) Support for a Regular meetings, Strengthened evaluation 6 regular start $50,000 GOG evaluation annual conference, community in Ghana meetings plus an 2000 2nd for three forum, with links to sharing of good- especially within GOG; with annual quarter; years only the fledgling practice evaluations links to the private sector, conference ongoing Ghanaian Evaluation universities, think-tanks and Association broader civil society (6) Creation of an 6 major evaluations Assist GOG in strategic Steering start $100,000 evaluation foundation per annum of decision-making committee 2000 3rd per to identify and strategic GOG Enhance accountability of comprising key quarter annum, commission selected programs or activities public sector to Ghanaian agencies and for three strategic evaluations society ministries, think years only Support GOG in its ongoing tanks, civil management of activities society and Establish best-practice private sector examples of evaluation (7) Broader support Training in basic Joint CIDA-World 2000 $50,000 in for civil society capacities in Bank training 2nd 2000; communication, course quarter $25,000 fundraising, policy p.a. for advocacy, the Additional 2000 next two building of alliances, courses offered 3rd years budget analysis, and periodically quarter M&E Provide NGOs and parliamentarians with access to M&E training courses 34 TABLE OF SUMMARY FINANCIAL COSTS Cost of #1: M&E #2: PPMED #3: Ministry #4: M&E #5: Ghana #6: Eval'n #7: Civil TOTAL Capacity- champion reviews mentoring training evaluation Foundation society building association Activity ($) 2000 30,000 141,000 66,000 150,000 (b) 50,000 (a) 50,000 (a) 50,000 (b) 537,000 Ongoing 30,000 131,000(a) 150,000 (b) 100,000 25,000 (b) 436,000 annual cost (a)continues for three years (b)indicative only 35 ANNEX B REVIEWS OF PPMED FUNCTIONS IN SELECTED GOG MINISTRIES GENERIC TERMS OF REFERENCE Background Most sector ministries contain Policy, Planning, M&E Departments (PPMEDs), and all are required by law to do so. PPMEDs typically play a central role in collecting and coordinating data within the ministry and from the departments and agencies which the ministry oversights (MDA ministry, departments and agencies), and in planning and coordinating the preparation of budget bids from the various parts of the ministry. The size and functions of PPMEDs appear to vary widely. Some encompass statistical functions; in others, such functions are performed by separate units. Senior officials and donor representatives have stated their belief that the work focus of PPMEDs tends to be bottom-up i.e., project and activity based, and principally concerned with the monitoring of financial inputs and of activities. While PPMEDs typically have some capacity to collect primary data, their ability to make good use of these data or to support ministry management's use of them, or to take a sector-wide approach, is unclear. Officials and donors have noted a need to strengthen PPMEDs, given their actual or potential central role within sector ministries, and given sector ministries' key role in sector planning and management. The priority for ensuring that PPMED functions operate efficiently and effectively has been enhanced by the opportunities and challenges of the Comprehensive Development Framework, and also by the related emphasis on identifying effective poverty-reduction strategies. The government's draft CDF emphasizes the priority in ensuring that a sound M&E system is in place; it also notes deficiencies in the approaches of the PPMEDs. Terms of Reference · identify the current range of functions of the PPMED and of related units (especially statistics units) · types and frequency of information collected and of analyses and evaluations undertaken (including, for example, budgeting, policy analysis and beneficiary assessment surveys) · map the extent of dissemination of this information/analysis · estimate the cost of each type of activity · identify users of this information/analysis, and the uses to which the information/analysis is put i.e., for purposes of management, planning, resource allocation and accountability · minister's office, senior management, other areas of the MDA including at the regional and district levels, NDPC, MOF, district assemblies, other line ministries, Ghana Statistical Service, civil society including representatives of beneficiaries, private sector, donors 36 · conduct a workshop to obtain the views of these and other stakeholders concerning the relative priority of the information/analysis generated · identify data gaps and weaknesses · identify good-practice examples, from which other ministries could learn, of M&E practices in areas such as coordination, planning, management, data collection, conduct of evaluation, analysis and reporting · identify the types of information/analysis which actual or potential users would prefer to have, and assess the strength of demand · i.e., who wants what, why, when · identify the cost of providing this desired information/analysis · identify the extent of demand under two alternative pricing regimes: (i) if the information/analysis was provided at no charge; (ii) if it was provided under a regime of full cost recovery · identify extent to which the MDA's needs for information/analysis could be satisfied from external sources such as GSS, universities and think tanks; identify the contribution which the MDA's data collection activities currently provide to national data collections · identify and make recommendations concerning the highest priority types and extent of information/analysis which the PPMED should provide or commission · identify indicative resource levels necessary to satisfy alternative levels of PPMED functionality numbers and skills of staff; access to consultant services; computer system costs; other costs · outline relevant database design and management issues, including data storage, processing and quality assurance · develop an action plan for in-house (e.g. recruitment, training) and/or outsourced functions · recommend appropriate management, reporting and coordination arrangements for these PPMED functions. This should encompass planning arrangements for deciding which MDA activities should be evaluated and when · examine whether it would be desirable to put in place service level agreements between the PPMED and the principal users of the information/analysis provided to cover dimensions such as frequency, timeliness, quality, reliability, objectivity and completeness · identify ways of ensuring the accuracy, objectivity and reliability of ministry data ANNEX C 37 TRAINING IN M&E The M&E courses would be organized in modules, designed to impart different levels of understanding and expertise, and targeted to specific audiences (see following matrix). A first introductory level will present the main issues concerning the evaluation topic: uses, strengths, weaknesses, possible abuses. This level provides awareness-raising. A second level would be offered to those involved with the phases of planning, design, or management. A third specialized level would be offered to the evaluation `doers'. The proposed strategy is to wholesale M&E courses to partner institutions in west Africa (such as the African Development Bank and the Joint Africa Institute) and in Ghana (such as GIMPA). Trainers from potential partner institutions would therefore be a particularly important audience. Emphasis would also be placed on reinforcing the capacities of Ghanaian think tanks which already have some specialization in evaluation and in development. Note that some of these modules could be delivered via their inclusion in other training activities. For example, a module on logframe and performance monitoring indicators could usefully be included in training provided via the Finance ministry in support of the medium- term expenditure framework. Topics/ Senior Technical level NGO's, think- Ministers, Trainers of Modalities of Audience government staff (doers) in a tanks, civil parliamentarians, partner delivery officials , high choice of society journalists institutions (who, level ministry ministries where...) staff Awareness Level 1 Level 1 Level 1 Level 1 raising, introduction to evaluation Training at all To be Logframe and Level 2 Level 3 Focus on service Level 1 levels determined performance delivery (Level 2) monitoring and budget indicators analysis Level 1 to 3 (think-tanks) Rapid review Level 1 Level 3 Think-tanks: Level techniques 3 Participatory/ Level 1 Level 3 Level 2 to Level 3 Level 1 stakeholder evaluation 38 Topics/ Senior Technical level NGO's, think- Ministers, Trainers of Modalities of Audience government staff (doers) in a tanks, civil parliamentarians, partner delivery officials , high choice of society journalists institutions (who, level ministry ministries where...) staff Cost/benefit Level 2 Level 3 Level 1 Level 1 analysis Program Level 1 Level 2 Level 2 to Level 3 Level 1 evaluation (management of (evaluation program professionals) evaluation) Training at all To be Policy Level 2 Level 3 Level 1 (NGO, civil Level 1 levels determined analysis/sector society) to level 3 reviews (think-tanks) Data analysis/ If needed, Level 3 Level 1 statistical for PPMED/ analysis statistics units Results based Level 2 Level 3 Focus on service management delivery (level 2) and budget analysis Level 1 to 3 (think-tanks) Level 1: introductory, in order to understand main issues: uses, strengths, weaknesses, possible abuses. Level 2: Intermediate, being able to contribute to the planning, design and management. Level 3: advanced, for doers.