57526 ECD Working Paper Series No. 18 Insider Insights: Building a Results-Based Management and Evaluation System in Colombia Manuel Fernando Castro www.worldbank.org/ieg/ecd November 2008 The World Bank Washington, DC Copyright 2008 Independent Evaluation Group Knowledge Programs and Evaluation Capacity Development (IEGKE) Email: ieg@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Building monitoring and evaluation systems helps strengthen governance in countries--by improving transparency, by strengthening accountability relationships, and by building a performance culture within governments to support better policymaking, budget decision-making, and management. A related area of focus is civil society, which can play a catalytic role through provision of assessments of government performance. IEG aims to identify and help develop good-practice approaches in countries, and to share the growing body of experience with such work. The IEG Working Paper series disseminates the findings of work in progress to encourage the exchange of ideas about enhancing development effectiveness through evaluation. An objective of the series is to get the findings out quickly, even if the presentations are somewhat informal. The findings, interpretations, opinions, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the Independent Evaluation Group or any other unit of the World Bank, its Executive Directors, or the countries they represent. ISBN 13: 978-1-60244-100-2 ISBN 10: 1-60244-100-6 CONTENTS Abbreviations ...................................................................................................... iv Foreword ............................................................................................................ v Executive Summary ............................................................................................. vi Introduction ..................................................................................................................................... 1 1. Evolution of the Monitoring & Evaluation System ............................................... 2 1.1. Institutional Context ................................................................................. 2 1.2. Implementing SINERGIA (1991­2001): Main Achievements and Obstacles ..... 5 1.3. SINERGIA and the Public Administration Renewal Program (2002­2006) ....... 7 2. Architecture and Main Tools of the M&E System ............................................................. 9 2.1. Institutional Base and Operating Arrangements ............................................ 9 2.2. M&E Components: Results-Based Monitoring ....................................................... 10 2.3. M&E Components: Results-Based Evaluations ........................................... 15 2.4. M&E Components: Results Dissemination and Accountability ..................... 18 3. Success Factors and Obstacles ......................................................................... 21 3.1. Role of the Department of National Planning as System Leader .......................... 21 3.2. Constitutional and Legal Support ............................................................................... 22 3.3. Role of the President's Office .................................................................................... 23 3.4. Financing Strategy and Staffing .................................................................................. 23 3.5. Methodologies for Defining Indicators ..................................................................... 25 3.6. Availability and Quality of Performance Information............................................. 26 3.7. Inter-Sector Evaluation Committee ........................................................................... 26 3.8. Regional and Sector M&E Pilot Systems .................................................................. 28 4. Recent Developments ........................................................................................................... 29 5. Lessons Learned and Conclusion ..................................................................... 30 Bibliography....................................................................................................... 33 Annex 1: Evolution of SINERGIA's Legal Framework ........................................... 35 Annex 2: SINERGIA Evaluations, 2002­2008 ....................................................... 36 iii ABBREVIATIONS BPIN National Investment Projects Databank (Banco de Proyectos de Inversion Nacional) CGR General Comptroller's Office (Contraloria General de la Republica) COINFO Inter-Sectoral Committee for Information Policy and Management (Comision Intersectorial de Politicas y Gestion de Informacion) CONPES National Council for Social and Economic Policy (Consejo Nacional de Política Económica y Social) DANE National Statistics Administration Department (Departamento Administrativo Nacional de Estadística) DDTS Subnational Sustainable Development Directorate (Direccion de Desarrollo Territorial Sostenible) DEPP Public Policy Evaluation Directorate (Direccion de Evaluacion de Politicas Publicas) DGPN National General Budget Directorate (Direccion General de Presupuesto Nacional) DIFP Public Investment and Finance Directorate (Direccion de Inversiones y Finanzas Publicas) DNP Department of National Planning (Departamento Nacional de Planeación) ICBF Colombian Institute for Family Welfare (Instituto Colombiano de Bienestar Familiar) INVIAS National Roads Institute (Instituto Nacional de Vias) M&E Monitoring and Evaluation MEN Ministry of Education (Ministerio de Educacion Nacional) MoF Ministry of Finance MPS Ministry of Social Protection (Ministerio de Proteccion Social) NDP National Development Plan PIPR Results-Based Investment Budget (Presupuesto de Inversion por Resultados). PMAFP Public Financial Management Program (Programa de Modernizacion de la Administracion Financiera Publica) POAI Annual Operational Investment Plans (Plan Operativo Annual de Inversiones) PRAP Public Administration Renewal Program (Programa de Renovacion de la Administracion Publica). RAS Social Safety Network (Red de Apoyo Social) SIGOB Government Results Information System (Sistema de Informacion de la Gestion de Gobierno) SINERGIA National Results-Based Management and Evaluation System (Sistema Nacional de Evaluación y Gestión por Resultados) iv FOREWORD As part of its activities, the World Bank's Independent Evaluation Group (IEG) provides technical assistance to member developing countries for designing and implementing effective monitoring and evaluation (M&E) systems and for strengthening government evaluation capacities as an important part of sound governance. IEG prepares resource materials with case studies demonstrating good or promising practices, which other countries can refer to or adapt to suit their own particular circumstances (http://www.worldbank.org/ieg/ecd). World Bank support to strengthen M&E systems in Latin America and the Caribbean Region has grown substantially in the past decade. There is intense activity on M&E issues in more than 20 countries in the region, and IEG has provided active support for many of them. In the case of Colombia, IEG has been advising the government and World Bank units, particularly since 2002, on ways to further strengthen the M&E system--SINERGIA-- with the objective of fully institutionalizing it. While different external assessments have been done on the strengths and weaknesses of SINERGIA and other Latin American M&E experiences, relatively few analyses have looked at the political economy of managing and implementing this comprehensive, government-wide M&E system, including the institutional, operational, and technical success factors and obstacles encountered. This Evaluation Capacity Development paper seeks to complement previous external diagnostics of the system with insights and perspectives from a former manager of SINERGIA: Manuel Fernando Castro, Director of Public Policy Evaluation, Department of National Planning. It is hoped that the lessons and best practices identified here, with regard to success factors and common obstacles, will benefit officials undertaking similar tasks in other countries. This paper has benefited from comments received from a number of people, including Keith Mackay, Gladys Lopez-Acevedo, and Nidhi Khatri. The technical assistance of Felipe Castro is gratefully acknowledged. Helen Chin contributed valuable help by thoroughly editing the paper. The views expressed in this document are solely those of the author, and do not necessarily represent the views of the World Bank or of the government of Colombia. Klaus Tilmes Manager Knowledge Programs and Evaluation Capacity Development v EXECUTIVE SUMMARY During the past two decades Colombia has been developing and improving its National Results-Based Management and Evaluation System (SINERGIA). The development of this system has been a focal point in Colombia's state reform towards performance-based management, particularly at the central administration. After 15 years of progress, overcoming the effects of institutional, political, and fiscal obstacles, SINERGIA has achieved one of the highest levels of development and customization, and it is held up as an example of best practices by multilateral organizations, donor agencies, and other governments. The monitoring and evaluation (M&E) system that Colombia adopted sought to promote a results focus on both planning and budgeting processes, simultaneously. However, during implementation the results-based planning model superseded the budgeting model, owing principally to the architecture of Colombia's central administration and to the institutional placement of SINERGIA within the Department of National Planning. Over time, various institutional solutions for conducting evaluations, strong advocacy by a powerful central department, and high-level political incentives have been needed to overcome problems between the planning and budgeting agencies--particularly regarding the use of results information and integration. Compared with similar experiences in other countries, the process by which SINERGIA advanced institutionalization is notable for the way it combined high-ranking, wide- ranging, formal mechanisms (e.g., constitutional mandate and laws covering the whole of public administration) with the development of informal practices in key areas of the public sector (e.g., M&E activity in planning offices, program management units, and managerial controls in the President's Office). Using lessons learned from international experience, Colombian officials followed an implementation strategy that--in line with the political and institutional reforms in the country--brought together a uniform, progressive approach for the central administration and the operation of selective, demand-based, pilot programs at the regional and sector levels. Many lessons for other countries are identified in this paper. Some of the key factors in Colombia's success are as follows: · The importance of having a central governmental department act as a champion supporter, and using an opportunistic approach to include the M&E system as a key pillar on public sector reform agendas; · The kind of regulatory and incentives framework on which the M&E system is based, and the sustained effort in promoting "cultural" change and developing evaluation capacities; · The political role of the President in the system's institutionalization process; · The powerful part that technically defined methodologies and dissemination mechanisms have played in the system's institutionalization strategy; · The collaborative approach used to strengthen evaluation practices and to enhance the community of evaluators; and · The demand-driven approach for introducing M&E at the regional level. SINERGIA did not develop in a linear, methodical manner to its current state of integration. It instead went through periods of rapid progress, stagnation, and setbacks in vi response to different contexts, as well as to changing political and economic environments. Following a contentious results-oriented public sector reform introduced by first President Alvaro Uribe administration between 2002 and 2006, subsequent changes in the institutional setting have tempered the momentum of SINERGIA. Although reforms introduced during Uribe administration resulted in the general public being widely approving of results-based M&E, some voices from academia and the private sector have questioned how independent SINERGIA is, and how credible the information it provides can be since it entirely depends on the executive. In addition, some of the system's institutional arrangements have begun to show its disadvantages, owing principally to the system's limited autonomy, funds, capabilities, and powers to regulate the evaluation market. In the context of a system that has encouraged a major change toward a results orientation in government, this type of external scrutiny should also be seen as an indication of the extent to which SINERGIA products (monitoring data, evaluations, reports, etc.) are used, as well as a sign of the progress made toward institutionalizing M&E in the public sector. Accordingly, efforts being planned to reinforce the quality, reliability, and credibility of M&E information--mainly through independent audits, academic analyses, and external evaluations--although critically needed at this time would strictly be of a short-term nature. In coming years, any real attempt to strengthen the system, and to consolidate the public sector's focus on results, will require further institutional re-design to tackle its current limitations. Recent international experiences in creating national evaluation commissions (e.g., Spain and Mexico) have shown that to institutionalize evaluation as a permanent high-quality practice, with objective standards, requires a guiding body with greater autonomy to regulate the evaluation market. An institutional change of this kind will, of course, entail further legal changes in Colombia, for which support from government and non- government sectors remains a challenge. The analysis presented in this paper shows some of the work carried out by Colombia, since 1991, to implement a national M&E system as part of public sector reform. Despite significant effort having been made, it is stated that much more work needs to be done if Colombia is to continue on the path toward full M&E institutionalization in government. Given the uneven manner in which SINERGIA has evolved, Colombia could further strengthen the foundations of its system by capitalizing what has been gained through broader institutional reform. What is required is a system that: depends less on the immediate extent of high-level support, is more autonomous and higher ranking, and is based on more powerful incentives. This paper suggests that accomplishing such a goal may require a somewhat different institutional architecture. vii viii Introduction Colombia's National Results Evaluation System (SINERGIA) is one of Latin America's outstanding results-based monitoring and evaluation initiatives. After 15 years of progress and after overcoming the effects of institutional, political, and fiscal problems, SINERGIA has achieved one of the highest levels of development and customization, and it is held up as an example by multilateral organizations, donor agencies, and other governments.1 As in other countries, the development of such a system was marked by periods of progress, stagnation, and setbacks. During the 1990s and early 2000s, state democratization and modernization processes--coupled with growing dissatisfaction with the results of government programs--were driving forces behind SINERGIA's implementation. By contrast, fiscal imbalances, acute budgetary restrictions, lack of integration between budgetary and planning processes, and a changing political and economic environment were all, in turn, opposing forces that led to stagnation and setbacks in the system. What differentiates Colombia's monitoring and evaluation (M&E) system from other experiences of its kind is that it has succeeded in integrating performance measurement as a current practice within all government planning frameworks. It has also succeeded in using results to progressively inform decisions and strengthen the accountability of national programs and sector policies. The SINERGIA experience in Colombia makes an interesting case study, for at least three reasons: (i) It illustrates one path among many that developing countries can follow for institutionalizing their own M&E systems. (ii) It establishes a results-oriented planning model, as opposed to the alternative and widely promoted budget-orientated approach. The planning model is important because, contrary to what is often thought, large numbers of countries, sectors, or agencies boast major strategic planning frameworks to guide government activities. (iii) It illustrates the extent to which an M&E system is not simply an isolated exercise, but is instead something that goes hand-in-hand with systemic public sector reform, affecting all levels of administration, involving the review of powers and responsibilities in different areas, and influencing the very architecture of the public sector. This paper examines the experience of institutionalizing Colombia's M&E system, from the perspective of a former SINERGIA manager: Manuel Fernando Castro, Director of Public Policy Evaluation, Department of National Planning. This study is meant to complement previous studies by other authors with an insider's account, description, and analysis of the obstacles encountered and the results obtained. Likewise, it presents recommendations on how to ensure further institutionalization and identifies several lessons that might be of interest to professionals who are developing evaluation systems in other countries. 1 See World Bank and IDB 2006; Guerrero 1999; Ospina and Ochoa 2003; Zaltsman 2006; Mackay 2007; and Villarreal 2007. 1 1. Evolution of the Monitoring & Evaluation System The origin of Colombia's National Results-Based Management and Evaluation System (SINERGIA) dates back to the early 1990s.2 The system was initially a strategic response to the need for results-based M&E in national planning, but it subsequently gained political and legal endorsement because of two major issues in the national debate. The first was the deteriorating credibility of public institutions in people's views, which was primarily a result of these institutions not being able to deliver results for the most important social priorities. The second issue had to do with high-profile cases of inefficiency and corruption in several investment projects, which indicated a need to improve transparency and accountability in government activities.3 Awareness by top-ranking officials about the political benefits of measuring performance4 was initially prompted by the efficiency concerns of a central government agency--the Department of National Planning (DNP). Such political concerns put M&E on the legislative agenda, and eventually led to its full legal incorporation into the national Constitution. In the process of developing these mandates, SINERGIA's design, implementation, and reform were based on international best practices, as well as on its own innovations and adaptations.5 1.1. Institutional Context Expanding the role of M&E has been one of the issues of far-reaching public sector reform in Colombia for a long time, particularly with regard to the government's planning and budgeting systems and their integration as government functions. Planning has a long tradition in Colombia. At least since 1949,6 but formally since 1991, development plans have been employed by both civil servants and elected officials to define policies, strategies, goals, and concrete priorities in public management. Development plans and, specifically, Annual Operational Investment Plans (POAI),7 are also the mechanism by which national, provincial, and local governments set up their investment budgets. In Colombia's administrative architecture, investment plans are prepared by the DNP (or by provincial/local planning secretariats), but current expenditures are set up by the Ministry of Finance (or by provincial/local finance secretariats). For this reason, at the national level (central administration) the planning and finance agencies each have their own individual budget office--namely, the National General Budget Directorate (DGPN)8 within the Ministry of Finance (MoF), and the Public Investment and Finance Directorate (DIFP)9 within the DNP. However, it is the MoF that has overall 2 SINGERIA was formally mandated in Colombia's Constitution in 1991, and implementation began in 1994. See Colombia's Political Constitution, Articles 339­344, and Law 152 of 1994. 3 The construction of Metro de Medellin and the hydroelectric dam in Guavio, two major projects at the time, presented problems of this kind. 4 Various high-level national and international seminars and conferences, between 1989 and 1991, played a major role in creating momentum and stimulating awareness by top-ranking officials and politicians. 5 Models studied included Australia, Brazil, Canada, Chile, France, Mexico, New Zealand, the United Kingdom, and the United States. At the local level, the budget-by-results examples of Sunnyvale (California), Austin (Texas), and Bogotá, in Colombia have also been important references. 6 The World Bank supported the introduction of a ten-year planning framework in 1949, which was changed to four years, beginning in 1962, to coincide with administrative periods. See Currie 1984. 7 Plan Operativo Anual de Inversiones. 8 Dirección General de Presupuesto Nacional. 9 Direccion de Inversiones y Finanzas Publicas. 2 responsibility for the government budget. The ministry has to integrate both budgets--its own and the Department of National Planning's--into a single document, and it is responsible for presenting it to Congress for approval. Coordination among finance and planning offices is therefore absolutely required. It is important to understand several other things about how M&E has evolved in Colombia: (i) Although the national M&E system was envisioned as a instrument for reinforcing strategic planning and investment allocations, it was also seen as a tool for pursuing political objectives, such as increasing state credibility, progressing toward a more democratic system, and achieving a more modern and transparent public administration. (ii) Since 1991, the National Planning System has had the same constitutional and legal standing as the National Budget System. As such, M&E received significant normative support. Colombia's legal tradition meant that a number of legal and policy instruments had to be in place to allow SINERGIA to develop. As a result, the system has highly legal (formal) institutional characteristics (see annex 1). Legal mandates were followed by effective informal M&E practices, which progressively gained acceptance in centralized and decentralized bureaucracies.10 (iii) The institutional framework for M&E also includes planning and evaluation units at the different levels of government, and the Public Policy Evaluation Directorate (DEPP)11 is the coordination agency at the national level. This unit was created in 1992 to be the secretariat and to be responsible for the design and implementation of SINERGIA, on behalf of DNP. (iv) The system was set up after a number of prior steps had been taken to implement ex-ante evaluation techniques in public investment projects. Worthy of mention among these are the Investment Project Monitoring and Evaluation System and the National Investment Projects Databank (BPIN),12 which improved investment programming and control processes, and helped to establish a budgeting-by-project culture. Because investment projects are the basic unit for preparing plans and investment budgets, having a national projects databank allows the National Planning System to have a detailed record of potential projects to be funded annually. It also permits ex-ante evaluations for determining the extent to which the objectives, activities, and resources required for such projects would also contribute to the wider purposes of national plans. (v) To the above points should be added existing processes and tools for planning and budgeting, which include mainly a financial plan, a multiple-year investment plan, and the Annual Operational Investment Plans at the national level. Moreover, there are sector-level action plans and strategic plans for individual agencies (see figure 1). These instruments are also used at the subnational and local levels and DNP provides technical assistance to those governments throughout the Subnational Sustainable Development Directorate (DDTS).13 10 Formal institutionalization refers to the fact that legal mandates, state policy guidelines, administrative technical procedures, etc., exist. Likewise, informal institutionalization refers to effective practices and the use of M&E tools, denoting acceptance by agencies and officials. 11 Dirección de Evaluación de Políticas Públicas. 12 Banco de Proyectos de Inversion Nacional, is under the DIFP at DNP. 13 Direccion de Desarrollo Territorial Sostenible. 3 Knowing about key features of Colombia's National Budget System is also central to understanding the institutional context in which the M&E system evolved. The budgeting system, under the guidance of the MoF, has coexisted, and to a certain extent even competed, with the equally important planning system. Although the law stipulates that the National Budget and the National Development Plan (NDP) should be complementary, the responsible agencies have not fully consolidated the necessary coordination processes. According to several analyses, the split between the current spending and investment budgets has restricted harmonization and limited full implementation of results-based M&E within the overall government budget.14 Figure 1. Colombia: National Planning and National Budget Structure of National, Sector, and Agency Plans Strategies and Programs and Objectives Policies Projects General part Annual and multi-year investments · Diagnosis · Earmarked social investment · Policy guidelines · New investment · Indicative plans: Goals and indicators · Financial indicators · Action plans · Results indicators National Budget Composition National Department of National - Investment Budget Investment Budget Planning Department Planning National Planning Statute National Budget Statute Ministry ofof Finance Finance - Currentbudget Current budget - Income budget Lending - Lending Income budget The link between the National Development Plan (which is the responsibility of sectors and agencies) and the National Budget is given by the goals and indicators established in sector and agency (four-year) strategic plans. The mission, vision, and objectives of the strategic plans are aligned with national development objectives and set the framework for the formulation of specific projects, which are registered in a National Investment Projects Databank. Accordingly, the model establishes that sector and agency strategic plans, the concrete projects contained in them, and the respective Action Plans to implement them must be the basic instruments for the preparation of the Annual Budget. Based on this planning structure, evaluation of results should take place within SINERGIA. Annual reports on plan performance must be presented to Congress. 14 For the implications of budget separation in Colombia, see World Bank 2005 and International Monetary Fund 2005. 4 In addition, given multiple regulations that pre-allocate expenses, the overall budget system has historically had high levels of inertia and rigidity (more than 67 percent in investments and 90 percent in operations) and low levels of accountability. For budget officials, rigidity has resulted in weak incentives for focusing on results, at best, and in strong resistance toward any changes in the budgeting processes, at worst. Some of this resistance also has to do with the historically little power that Congress has exerted over the budget and also with the relatively high discretion of the MoF. Given these circumstances, it is not surprising that during a period of major fiscal constraints, the original 1989 Budget Statute, Law 38, did not include evaluation of expenditure in any of its 127 articles. Despite numerous amendments for establishing greater spending controls to ensure fiscal discipline, there was never any fundamental change to the statute. The legislative record gives an indication of the little interest that budget officials had in results-based M&E of the budget for a long time.15 This lack of interest was in strikingly marked contrast to the importance given, historically, to the results-based approach by planning officials. 1.2. Implementing SINERGIA (1991­2001): Main Achievements and Obstacles In its first stage, SINERGIA introduced a self-evaluation approach based on adaptations of the logical framework's methodology--from the program to the sector and agency levels--primarily using monitoring information. This methodology was called the Plan Indicativo, and it helped to organize the management of sectors and agencies, in line with broader strategic government policies. In the Plan Indicativo each sector and agency identified its objectives, activities, targets, and performance indicators and established linkages among investment projects, sector and agency programs, strategic plans, and the final objectives of the NDP.16 To complement this, SINERGIA also introduced "strategic evaluations" in pilot agencies. Other instruments implemented included efficiency agreements, performance contracts, and commitment policy documents approved by the National Council for Economic and Social Policy (CONPES).17 All of these instruments incorporated specific coordination arrangements between the responsible agency and the DNP for achieving the goals set under the plan.18 A self assessment of SINERGIA for the period 1991­2001 identified the following achievements.19 Indicators and goals were introduced and produced on a regular basis in 19 sectors, in more than 250 central government agencies, and in the country's 32 provinces. 15 In 1999, the state reform plans of the current administration sought to dismantle the system as a contribution to austerity measures, but the system survived because of its Constitutional mandate. 16 See World Bank 1997; Ospina and Ochoa 2003; and DNP 2005. 17 CONPES is the highest collaborative authority for policymaking in Colombia. It is headed by the President and is composed of the Ministers and Central Bank President. 18 These coordination mechanisms took place only at technical levels and, unfortunately, not among higher- ranking officials. It included fifth-level echelons from the implementing agencies, the sector units at DNP, and SINERGIA (see the 2004 document, CONPES 3294). 19 The diagnosis set out to identify strengths and weaknesses and to appraise instruments, the regulatory framework, and the professional and financial skills that were available in the technical unit and in the different entities and sectors. It also included an analysis of evaluations undertaken, reports, and M&E information generated by the system. It likewise included interviews at the political level, with Ministries (Ministers and Deputy Ministers), at the technical level (planning and budget directors), and at the operational level (persons responsible for executing programs). 5 The system developed a methodology for monitoring and evaluating national plans as well as sector and agency strategic plans. It also established the practice of producing annual reports that showed results achieved under such plans. SINERGIA introduced basic instruments for performing M&E activities and promoted performance measurement "practices" in central government agencies and certain local authorities, especially planning offices. The above achievements set up important milestones for the public sector modernization process because they established the bases for the government to start moving toward results-based management. However, the M&E system did not yet significantly affect decision-making processes, particularly, policy formulation or resource allocations. Neither was it yet a mechanism for accountability to higher managerial levels, to Congress, or to the general public. Analysis revealed different findings related to the above outcomes during SINERGIA's first phase: Between 1998 and 2002, the system engaged in measuring the results of 32 provincial departments and 1,099 municipal development plans.20 The capacities of SINERGIA's Public Policy Evaluation Directorate (DEPP) unit were largely exceeded by this task.21 Moreover, the scope of the system's activities went beyond its mandate, due to overlap with the fiscal, political, and administrative functions of regional governments, and thus exhausted the system's own capacity. Budgeting and planning diverged further and further over time even though, in theory, the budget should be based on the plan. The MoF did not consider plan results when preparing the budget, and, without performance information, the President's Office could not identify misalignments between government priorities and annual budget allocations. In 2002, for example, almost 40 percent of the annual budget did not reflect any government priority in the National Development Plan. M&E therefore only partially covered government expenditures. Given the functional separation between the budget and the plan, coordination solutions were only available at specialized levels, particularly among budget officials. The priority for such officials is aggregated expenditure controls at the macro level, not the use of incentives (rewards and sanctions) to stimulate good performance at a micro-managerial level. There were problems with performance information availability, quality, frequency, and access. In 2002 SINERGIA had a large number of indicators (946) but--in marked contrast to its conceptual focus in higher-level results--58 percent of them were related to operational activities and administrative processes.22 In addition, ministries and agencies were able to send their reports electronically, but central consolidation was not automatic. Information was not available in real time and was not made easily available to the public. Reports therefore failed to provide timely information. 20 Provincial governors and mayors are elected by popular vote in Colombia, and the provinces and towns are fiscally and administratively autonomous in how they use the transfers they receive from the central government. 21 In 2002, SINERGIA had 42 professionals, consisting of in-house officials, consultants, and administrative staff. 22 Twenty-one percent were input indicators, 37 percent were activities and processes, 29 percent were outputs, and 13 percent were outcomes and impacts. 6 In-depth evaluations (impact and other methods) were in the early stages of development, and only a very small portion of investment expenditure was being evaluated (4.1 percent in 2002). Different sectors carried out evaluations but there were no defined technical standards for determining what programs to evaluate, what methodologies to use, or what procedures to follow for guaranteeing the quality of both evaluators and their evaluations. Moreover, few qualified evaluators were available--especially in impact methodologies--which meant that the cost of studies requiring international experts was potentially high. There was little dissemination of results to decision-making bodies in government, to Congress, or to the general public. Many of the evaluations were known only to those who had commissioned the studies and, in many cases, not even the decision-making bodies in those agencies were aware of their existence. As far as monitoring was concerned, results were consolidated and presented annually or every six months (for exceptional circumstances), but day-to-day, performance information was not readily available to be of help in government administration. 1.3. SINERGIA and the Public Administration Renewal Program (2002­2006) In mid-2002 Colombia elected Alvaro Uribe into office as President, and his administration opened a window of opportunity for SINERGIA. The Public Administration Renewal Program (PRAP), 23 a cornerstone of the new government, introduced management by results, transparency, and accountability as key principles for public sector reform.24 SINERGIA was thus included in a number of crosscutting reforms of the PRAP: civil service, public budget, procurement, state regulation, and public information, among others. The political climate for reform can be clearly seen in Colombia's 2002­2006 National Development Plan approved by Congress: The overall panorama indicates a worrying deterioration in the credibility of public institutions and in people's trust in them, due particularly to a perception that they are inefficient in their performance of essential state duties connected with security, economic growth, and improving the living conditions of the poor. Moreover, public finances have witnessed a rapid growth in expenditure, a high fiscal deficit, and an explosive public indebtedness. All of this has resulted in a weakening of governance conditions and in governments being less able to successfully perform their duties. Against this background, the new President himself demanded that government be given tools to enable it to monitor and control how the administration was achieving its aims. He also had stressed the government's "obligation to provide prompt and reliable information about the results that have been obtained with public funds, and the people's entitlement to access that information."25 The new administration thus gave strong political weight to results, performance information, and permanent accountability. SINERGIA received new importance; it became the focal point of the results-based management reform. 23 Programa de Renovación de la Administración Pública. 24 According to the President's Directive No. 10 of 2002, the objectives of PRAP were to modernize public administration, to consolidate a managerial state with fiscal responsibility, and to guarantee that public services would be rendered on the basis of efficiency and quality, thereby combating corruption and re- establishing state credibility. 25 Álvaro Uribe Vélez, President of Colombia, 2002­2006 (see Vélez 2002). 7 The SINERGIA reform began in October 2002, but was not formally written up until almost one and a half years later, in 2004, as CONPES 3294. The redesign was headed up by the National Planning Department team in charge of PRAP, and specifically by the SINERGIA/DEEP team. One of the very first actions of reform was to divest SINERGIA of all activities outside of its core mandate, particularly at the provincial and local levels. The decision was made to focus the M&E system on 19 central government sectors and 250 agencies. It was hoped that SINERGIA could be a model for regional governments, leading to the replication and autonomous implementation of its tools. This narrowed focus on regional governments, together with the findings of the 2002 self-assessment facilitated better identification of SINERGIA's main clients and better adaptation of system tools to performance information needs. To complement its original objectives, SINERGIA specified an additional goal in 2002 to help improve the transparency and quality of public expenditure. The context for this goal was an unsustainable rising trend in expenditures which had begun in the early 1990s. Total expenditure as a proportion of GDP rose by 52.9 percent26 and the fiscal deficit consequently deteriorated to 6.1 percent of GDP between 1995 and 2002. Given the small number of evaluations undertaken, the results of these expenditures were largely unknown. The government endorsed the above PRAP approach, and the following SINERGIA objectives--which emphasize transparency and expenditure quality--were approved by CONPES:27 (i) To improve the availability and quality of performance information and make that information accessible for accountability and decision-making purposes; (ii) To establish a closer link between policy and program planning, budgeting, and execution processes, by making use of performance information; (iii) To introduce institutional coordination arrangements, in order to better align supply and demand of M&E information; (iv) To focus performance information on variables of interest to the President, decision-makers, and the general public, and to encourage use of this information through different dissemination mechanisms; and (v) To develop evaluation capacities and the availability of evaluators in the country, so that the quality, results, and impact of programs and expenditures can be measured at reasonable cost. By customizing the system to the decentralized architecture of the state, as well as to the political and technical goals of the new government PRAP, the reform introduced a fundamental change in SINERGIA'S strategy. As a complement, from 2003 to 2006, with support from national and international stakeholders,28 SINERGIA/DEPP organized high-level international conferences each year as well as seminars, workshops, and training courses. These events had a major impact on the national debate about the role evaluation 26 Estimates are based on appropriations for the central government by the General National Budget Directorate. 27 See CONPES 3294 of 2004. 28 Namely, the Ministry of Finance, Central Bank, World Bank, International Monetary Fund, and Inter- American Development Bank. 8 should play in public finances, increased the awareness and M&E capacities of selected trained public officials, and helped to showcase advances in results-based management reform. This level of interest helped to establish SINERGIA/DEPP as a leading organization in the debate on performance-based management and budgeting in the country.29 The forums generated a new national and international perception of SINERGIA's relevance. This perception even improved in many respects in the following years, and has made it much easier for the system to continue executing reform strategies. 2. Architecture and Main Tools of the M&E System The aim of the M&E system was defined conceptually right from the start in 1991 and has remained fundamentally unchanged, even after the 2002 reform. What the reform did modify were the operating arrangements, the outreach of the system's components, and the set of M&E methodologies seeking to bring the system into line with the new institutional context in Colombia. 2.1. Institutional Base and Operating Arrangements As previously mentioned, in its first stage SINERGIA did not significantly affect government decision-making processes; neither was it an effective accountability mechanism. Assessment of the system, carried out in 2002, showed that these aspects were closely linked to lack of coordination between SINERGIA and different political levels (the President, Congress, and general public). The institutional placement of SINERGIA within the Department of National Planning, rather than in the MoF, also played a role in limiting progress toward a systematic evaluation of spending through a results-based budgeting approach. New operational arrangements were introduced to improve coordination. These included the Council of Ministers and the CONPES, as high-level decision-making bodies, as well as the President's Senior Adviser in a political supervisory role of the whole-of- government's performance (figure 2). In addition, an Inter-Sectoral Committee for Evaluation and Management for Results was established. It consisted of the national planning and budget authorities, and representatives from the President's Office, the ministries, and agencies of the key programs to be evaluated. The committee was to be the principal inter-ministerial/agency coordination mechanism in the central government for ensuring integration and alignment among planning, budgeting, and evaluation functions. It was also an instrument for promoting all activity related to strengthening the results focus in government. The MoF's Budget Directorate assumed an active role on the evaluation committee, while the Department of National Planning (specifically the DEPP) assumed the Technical Secretariat.30 The secretariat DEPP became the support unit for the President's Office to carry out periodic performance management controls with ministers and agency directors. Finally, Congress, control agencies, and civil society organizations were explicitly identified as the key external users of the system's performance information. 29 From 2003 to 2006, SINERGIA organized an international seminar on the subject at least once a year, thereby guaranteeing continuity and an interest in expenditure evaluation issues, both within and outside government. 30 See CONPES 3294 of 2004. 9 Figure 2. SINERGIA Institutional Framework Congress Civil Society CONPES President / Council of Ministers Inter-Sector Evaluation DNP / Committee President's Public Policy Evaluation (President, MoF, Senior Adviser Directorate DNP, Ministries and invited Agencies) Ministers / Local Control Civil Society Agencies Governments Bodies 2.2. M&E Components: Results-Based Monitoring The instrumental and methodological changes introduced in the second stage of SINERGIA (2002­2006) sought to provide answers for different problems, although they were not necessarily successful in all cases. The starting point was the change from a self-evaluation approach to one of external evaluations, performed or contracted out by SINERGIA/DEPP. To this was added the explicit differentiation, both conceptually and operationally, of monitoring, evaluation, and accountability activities. This differentiation gave rise to the current components of the system. Each of these components, with different but strictly complementary mechanisms, required its own methodologies. But the conceptual differentiation was also carried out with a view to establishing a common language among the different stakeholders in the M&E system, thereby progressing toward institutionalization of M&E tools and functions. The actions taken and the problems of defining new instruments in each of these components are discussed below. 2.2.1. Government Results Information System (SIGOB) SINERGIA implemented a technological tool in 1996 for automating and simplifying the consolidation, administration, analysis, and reporting of results data. By 2002, this information software was obsolete because it required the manual loading of files submitted to DEPP, and it could not communicate with other systems. Information updates were therefore not regular, data did not cover all central government agencies, and consolidation was a major operational burden on Evaluation Directorate staff and system agencies. The 2002 change in government and the President's new results information requirements demanded that the existing tool be updated quickly. Because contracting for a new 10 technological platform would take several months--in addition to the time it would actually take to develop a new system--the government asked the United Nations Development Programme to donate their existing software known as the Government Results Information System (SIGOB). In its original version, SIGOB was a presidential goals scorecard, with most of the indicators associated with operational activities (meetings, tendering processes, communications, etc.). It had rigid information functions and a number of fixed rules for loading and consulting information. Accordingly, SINERGIA/DEPP had to define technical content that agreed with the higher-level strategic objectives of DNP, in order to ensure that quality and relevant performance data were produced. The DEPP team defined a new SIGOB information content, taking into consideration all of the President's information needs, the general public and Congress' accountability requirements, and relevant policy and budgetary decision-making information. Based on such DEPP definitions, information technology experts were hired to bring SIGOB technology into line with SINERGIA requirements. In practice, the modified SIGOB became an inter-institutional coordination mechanism, which made it necessary to define clear roles and responsibilities among the DNP, the President's Office, ministries, and agencies. Once this step was completed, line ministries and agencies were given access to SIGOB to enter their goals and indicators. Such information is supplied along with the respective investment budget. In addition, a manager--whose name is made public through the website--is assigned to each goal in order to strengthen accountability for the information reported. Based on these records, the DEPP team controls and validates information quality. A team of sector experts is responsible for maintaining dialogue with agencies as well as the goal managers in charge of loading information onto the system. The results-monitoring function thus began to operate as an integrated, online system, which included quality parameters and increasingly up-to-date data. By the end of 2005, the SIGOB database had become one of the main sources of information for the government. The ability to determine trends in results and to have an up-to-date indicator database led to growing demands from ministries and other entities, but especially from the President's Office, DNP, and the MoF. The statistics from the SIGOB database are used heavily by the President's Office in press releases and reports to Congress, in development plan evaluations and reports by the DNP, in Sector Ministers' meetings and reports, and in MoF budget documents submitted to Congress. Despite all of these improvements, the availability, timeliness, access, and quality of information still remains a challenge for SIGOB because no regular external data audits are done and problems of differing variables and sources still limit complete reliability. 2.2.2. Defining and Simplifying Goals and Indicators Contrary to what is generally thought, it is no easy matter to draw up goals and indicators. Techniques and a methodology are required, as well as clearly defined uses of the information. Prior to 2002, low relevance (mainly concentrated on the operational side) and too many indicators in a large number of sectors and entities had resulted in a vicious circle of bad quality (little relevance) and little usage of performance information. In order to deal with these problems, a standardized method was defined for all sectors in 2002. The aim was to strengthen the results orientation, and the method was based fundamentally on three premises: (i) focusing system indicators and goals on strategic DNP objectives, particularly stressing their relevance to the President, to citizens, and to decision-makers; (ii) 11 agreeing on the definition of indicators, goals, and baselines directly with those who are responsible for the programs, and assigning each goal or group of goals, depending on their importance, to a manager; and (iii) bringing ministers, managers, and the President's Office into the policy goals discussion and approval process. An overview of key steps for defining and monitoring indicators is provided in box 1. Box 1. Using Performance Indicators and Monitoring Information Step 1. Definition and selection of indicators A proposal is drawn up by DEPP staff and by sector/agency planning and budgeting offices. The proposal is submitted to ministers and agency directors. Step 2. Discussion and validation Discussion and agreements are undertaken at individual meetings with ministers and directors. Relevance of the respective goals and indicators is assessed, taking into consideration: (a) Funds allocated in the previous years, and actual performance (baseline); (b) Current allocations; (c) Technical and institutional capacities for reaching the goals; (d) Agencies' control over result variables; and (e) Potential restrictions to performance. Agreed NDP indicators and goals are presented to the President at a Council of Ministers for approval. Step 3. Disclosure: Publication of National Development Plan goals and Indicators booklet. Submission to Congress, universities, and the media for strengthening public commitment. Indicators loaded onto SIGOB website with names of the goal managers responsible for reporting. Step 4. Monitoring Goal managers record progress information, directly pointing out the factors determining performance. Public explanation provided for shortfalls in performance (town hall meetings and ministry councils). DEPP staff monitor indicators for identifying obstacles, interacting daily with programs responsible staff. The President holds periodic individual management controls with ministers and managers. Step 5. Reporting DEPP draws up progress reports every three months, selective monthly electronic reports and the President's annual accountability report to Congress. Establishing indicators of interest to the President and that reflected tangible government results produced a number of effects, at least in the following respects. The President, his more immediate advisers, and the different ministries became more active users of the system's results information, given its usefulness for official press releases and presidential speeches. Because there was an increasing demand for, and disclosure of, this information from the President, ministers and their officials at managerial and technical levels also began to demand it, so they could interact with their superiors. The NDP became an important focal point in discussions throughout the administration's period in office (four years), and budget officials began to take results information into account and to be interested in better integration between the development plan and the annual budget. The President directly asked the Budget Director to take government goals into account when establishing the allocation of funds. (Prior to 2002, attention to the NDP occurred mainly during administration's first year in office when it was discussed and approved by Congress. After the first year, it lost attention in government discussions, to the extent that many officials and analysts started to question its managerial utility.) Those responsible for drawing up indicators learned by doing the job and by working directly with the SINERGIA team; this in itself was a process for transferring skills. 12 Working jointly with the agencies and programs in defining performance indicators provided SINERGIA an additional mechanism for quality assurance. Assigning each goal to a manager, whose name was made public through the SIGOB website, meant that the quality of the information reported could be investigated more easily and incentives were established for reporting more accurate data.31 The process of defining goals and indicators was made uniform under a single methodology, and common technical criteria were established for all ministries and agencies. The government's annual and four-year goals were to be discussed by the Council of Ministers and the President. Another central aim of the new methodology for establishing goals was to reduce the number of indicators. As mentioned earlier, in August 2002, SINERGIA had 946 indicators, the majority of which had been determined on the basis of operational objectives (activities and inputs) rather than the expected outputs, outcomes, and impacts of the main programs.32 The problems in analyzing such a large number of variables made it difficult to establish monitoring priorities or to direct sector activities, thus making the system a less credible managerial tool. Furthermore, available information was incomplete. The number of indicators was therefore cut to fewer than 300, and the focus was placed on measuring outputs (61 percent), outcomes (27 percent), and impacts (12 percent). The process was carried out in a standardized way for all sectors and entities.33 The number of indicators was still large but, because they covered 19 central government sectors and all priority NDP programs, the database was more manageable and relevant for the system's M&E functions. As a result of this thinning-out process, results monitoring became easier for DEPP, as did the writing and dissemination of reports. Procedures for a simplified base of indicators were formalized in a methodology manual, which also set out common rules relating to minimum frequency and steps for updating information for performance monitoring purposes. Restrictions were also established on amending goals. All of this was done with a view toward establishing clear, sustainable rules relating to the formulation, administration, and use of performance indicators. 2.2.3. Management Controls with the President Clearly, one of the most powerful mechanisms for developing and consolidating the second stage of the M&E system has been the periodic progress review meetings held by the President with his ministers. These meetings are also attended by the deputy ministers, who are sometimes supported by their technical managers. SINERGIA/DEPP staff participates in all meetings and provide technical assistance. These results review meetings are very collegial and informal, which is reflective of the current President's personal results-oriented style. There is therefore some risk that the current arrangement will not continue when a new administration comes into office. However, the value of the progress review meetings has extended beyond the central government--many regional and local governors are already using similar mechanisms in their own administrations.34 31 Various instances of inaccurate information led to penalties, public warnings, and even dismissals of personnel. This got an important message across to others in charge about the seriousness of the exercise. 32 Fernández and González 2006. 33 Ibid. 34 Specific cases of this are the cities of Pasto and Medellín, but many other towns and provinces have developed their own versions of this mechanism. 13 2.2.4. Linking Results to Funding Starting in mid-2003, an effort was made to link information on actual government spending with NDP objectives, policies, and programs goals, mainly for accountability purposes. This approach--sometimes referred to as "presentational" performance budgeting--was introduced through the preparation of a Results-Based Investment Budget (PIPR) document.35 This document was meant to inform the general public and the government on the connection between goals and expenditures, thus making the budget more transparent. The PIPR showed the breakdown of the results obtained, compared against the previous year's budget, thus providing an idea of the minimum investment requirements needed to achieve equal or better results in subsequent years. It was also hoped that the effort would lead to more informed discussions in Congress. The first PIPR exercise was carried out in 2003, and was used exclusively for loading budget information onto SIGOB. But the second exercise was presented to the Council of Ministers in October 2004, as part of the discussion process for the proposed 2005 budget to be submitted to Congress. PIPR was also meant to help reduce demands for additional budget allocations by those sectors that had underperformed in reaching their goals, but it also showed the level of funding needed if goals were to be increased in subsequent years. The initial intention of SINERGIA/DEPP was to include the PIPR in the budget proposal to Congress, but the Budget Directorate at the MoF objected, arguing that doing so could create pressure on the government, by Congress, to spend more. Instead it was presented as an attachment to the budget document, after the budget was actually approved by Congress. The PIPR exercise had, in fact, a more structural but lesser-known effect. At the time, discussions were being held--as part of the State Renewal Program--on reforming the Budget Statute, and the PIPR's launch paved the way for including results-based evaluation practices during the budget preparation process into the budget law. The MoF's Budget Directorate asked SINERGIA/DEPP to prepare a chapter for the new law on evaluating expenditure, which would be included in the proposal that was to be submitted to Congress. DEPP produced an ambitious chapter, which was significantly shortened by budget officials. A number of essential aspects nonetheless remained. The reform bill was again rejected by Congress, and the government had to accept a lesser reform. Decree 4730 was enacted in 2005, and for the first time in Colombia's budget history, it included four articles central to moving toward a results orientation in public expenditure, specifically, the Introduction of results-based M&E activities as regular practices in the budget cycle (Article 5); Inclusion of a program classification of budgets as a new economic classification category for expenditures (Article 8); Use of impact evaluation findings under the standards set out by the DNP as a key input of the budget programming process (Article 9); and Institutionalization of the annual evaluations agenda that SINERGIA had established in CONPES 3294 of 2004 in the form of budget legislation (Article 34). 35 See Curristine 2006. 14 Based on the above, a new results-based monitoring and evaluation framework for budgeted resources was built around five stages: setting plans; defining indicators and goals; linking plans and budgets to results; monitoring and evaluating results; and presentational PIPR (figure 3). Figure 3. Results-Based Monitoring Phases and Involved Agents 1 National Development Plan: objectives, DNP strategies, and programs. Results-based Agencies Civil Society 2 Set up of indicators, goals, and Presidency ­ DNP baselines. Agencies Council of Ministers 3 Integration of budgetary allocations to set Presidency goals. Link between funds and goals. DNP - MoF 4 Physical and financial monitoring. Performance assesments. DNP - Agencies Results-based Investment Budget and MoF, DNP 5 other Accountability Reports 2.3. M&E Components: Results-Based Evaluations As discussed earlier, very few ex-post program evaluations were done during SINERGIA's first stage of implementation. Evaluations that were carried out were more in the realm of pilot experiences than an evaluation model, as such. Evaluations of the whole NDP are a category apart, because they follow a specific rapid methodology set up by SINERGIA, based on monitoring information. There was therefore a significant imbalance in the M&E system during the first stage, when it was skewed toward monitoring activities and, to a large extent, away from in-depth evaluations. SINERGIA had moved away from its mission: to operate as an evaluation system with coordinated, balanced M&E functions and methodologies. It needed to have a number of complementary tools to rely on, to meet specific knowledge demands about the results of policies and public spending. In addition to monitoring government policies, SINERGIA had to undertake or promote program-centered evaluations on a larger scale, and these needed to be strategic and conducted by external experts, to ensure independent and thus credible analyses. Taking all of these needs into account, and adhering to the most widely accepted methodologies, SINERGIA introduced new evaluation concepts and minimum standards between 2002 and 2006. Four new methods were added to the two types of evaluations that were already in use36 (table 1). Even with these new changes, the number of evaluations, though increasing, could still be considered low. This is especially so when one considers the total number of national programs, the proportion of the investment 36 See CONPES 3294 of 2004 for a wider description of such concepts and methods. 15 budget subject to evaluation, and, especially, the actual use of findings for policy-making or budgeting purposes (see annex 2). Table 1. Evaluation Methodologies of SINERGIA Type of Evaluation Measurement Object and Characteristics Initial 1994­2001 Analysis of activities, goals, administrative processes; use of necessary Operational or Process resources for achieving the proposed objectives. Appraisal of whether the objectives of the National Development Plan Results have been achieved in terms of products and outcome indicators. Additional 2002­2006 Causality analysis to determine effects of a program on beneficiaries Impact (expected or unexpected, direct or indirect). Comparison of social and economic variables to establish monetary Cost - Benefit values of whether, and to what extent, benefits are greater than costs. Assessment of inputs and outcomes to identify the extent to which Productivity institutional redesign changes the productivity of an agency. Standardized, rapid (3-month) analysis of design, operational, Executive institutional, and financial structure and M&E information. Source: SINERGIA. Operational arrangements for conducting evaluations were also introduced, primarily: (i) appointing a group of professionals at DEPP to conduct technically designed evaluations, thereby assuring quality standards; (ii) having a plan for contracting out evaluations that would ensure independent analysis while stimulating the development of evaluation skills; and (iii) putting in place an institutional mechanism for evaluation, to provide feedback for decision-making processes. Relevant background for these arrangements existed in Colombia in the impact evaluation plan of the Social Safety Network (RAS),37 which was established in 1999 by the DNP.38 This was, nonetheless, an ad-hoc impact evaluation scheme, which was set up in parallel with the existing SINERGIA/DEPP, and had little participation by relevant sector ministries. Another major reference was international experience, particularly in evaluating the impact of conditional cash-transfers programs in Mexico and Brazil. Following the evaluation experience of cash transfer programs in the above countries, the DNP had formed an impact evaluation team for drawing up and coordinating the evaluation of the newly introduced RAS. The SINERGIA reform in 2002 meant there was a need to combine the DNP teams responsible for coordinating national evaluation activities and the RAS impact evaluations. The latter's technical team was therefore transferred to the DEPP and efforts were made to extend evaluation activities to other programs with this newly combined scheme. 37 Red de Apoyo Social. 38 RAS was the first social protection network initiative in Colombia to be implemented with a view toward counteracting the effects of the 1999 economic crisis on the poorest populations. It was made up of three programs: "Empleo en Acción," which sought to provide the poorest people with temporary jobs, "Familias en Action," which set out monetary subsidies to improve nutrition and school attendance, and "Jóvenes en Acción," which aimed to improve job market access for the unemployed. RAS was partially funded through loans from the World Bank and Inter-American Development Bank. 16 The RAS team at DNP contracted out evaluations to ensure they were conducted impartially, and SINERGIA adopted this approach so that the findings and recommendations would be independent and credible. External evaluators--from prestigious national and international universities and firms--were selected based on their expertise and proposed evaluation methodologies; the estimated cost of the proposed evaluation was of secondary importance in the selection process. Another important practice SINERGIA instituted was to separate the functions of executing the programs from the contracting and supervision of evaluation activities. This practice brought several benefits: it allowed SINERGIA/DEPP to play an intermediary technical role between the programs and the evaluators, relieved the programs from additional operational burdens that evaluation on a permanent basis implies, and gave evaluators a full guarantee that programs would not exert any influence on their judgement when they conducted their evaluations. It also allowed a team of professionals with knowledge of and exclusive responsibility for evaluation to interact with the evaluators, while giving evaluation the priority it deserved. Programs are usually embedded and overly burdened with implementation activities; evaluation thus does not receive priority. Apart from the greater independence made possible by this external selection and contracting arrangement, SINERGIA also found a way to improve the development of evaluation capacities in the country. Contract arrangements involving joint efforts and alliances between highly qualified international evaluators and national research centers, firms, or universities were given higher scores in the procurement process. Knowledge- and technology-transfer processes were thus established, and in less than three years there were already Colombian firms capable of carrying out top-quality evaluations on their own. The cost of doing impact evaluations in the country also fell because local evaluators are typically less expensive than international ones, which additionally made it possible to perform more evaluations. Finally, in order to ensure that evaluation results could influence policy and budget decisions, DEPP created an Inter-Sector Evaluation Committee. This committee was initially made up exclusively of technical units from the DNP, but it soon became apparent that this was too limiting. In the 2002 reform, the DNP Director General extended the committee's membership to include the MoF, the ministries responsible for the programs, and the President's Office. A direct channel to different highest-level policy decision-making bodies in government was established through this route while evaluation supply and demand were brought into line by matching the performance information needs of budget, planning, and sector officials with available or possible performance evaluations. Together with the discussion and analysis of findings from evaluations, this committee had the task of defining an annual evaluation agenda, which would include programs to be evaluated with the various methods defined by SINERGIA. An important indication of the extent of change the reform had on evaluation activities is the increase in the percentage of the nation's investment budget subject to some kind of evaluation: from 4.1 percent in 2002, to 23.4 percent in 2006. The individual number of evaluations carried out under the system also rose, from 3 between 1996 and 2002, to 23 between 2002 and 2008, Sixty-five percent of these (15) were impact evaluations, and 35 percent (8) were other types (see annex 2). The costs of these evaluations ranged from US$25,000 to US$1.5 million, depending on the type of evaluation and on the availability of performance information. 17 2.4. M&E Components: Results Dissemination and Accountability As a supplement to the M&E functions, a third component was built into the 2002 reform, with a view to institutionalizing dissemination channels and accountability methods. This component sought to ensure that the general public could access performance information; and it was hoped that this would put pressure on data providers to ensure information accuracy. Four instruments have been implemented in this field, which were either directly developed by the system or implemented by the government, acting on the President's instructions. These instruments played a major role in the second stage of the M&E system. 2.4.1. Annual Report by the President to Congress Prior to 2002, annual President's reports to Congress were a mere formality. They were a compilation of unrelated documents that had been submitted by the different ministries to the President's Office before being sent on to Congress every July. The DNP, following the President's guidelines, set out to restore the value of accountability to Congress and to give this function greater importance within the government. The understanding was that accountability would contribute toward improving governance and transparency while promoting good practices in strengthening political controls. Right from the start of the SINERGIA reform, DEPP identified the annual report to Congress as one of the system's central products, and began to redesign it. The re-design sought to show: Linkages between election proposals and government goals, as set out in the NDP; Sequential assessments of the state of the country, starting with a review of the past, moving on to the current state of the country, and concluding with where the country is headed, in terms of prospects and goals; Review of both achievements and setbacks; Reader-friendly presentations, using simple language, together with as many graphic elements as possible to illustrate sequences; A short document accompanied by an executive summary or overview; and Consistent information and widespread dissemination. Since the first version of the report was published in July 2003, it has been eagerly awaited each year.39 The media disseminates sections of the content, especially administrative flaws, but several opinion journalists have been generally more evenly balanced. The President's communications office disseminates the content widely, as part of its official communications strategy. Consistent use of the information and a planned dissemination strategy by the system's DEPP technical team significantly contributed to its wide circulation. The user-friendly presentations and simplicity of the data and results are important in effectively conveying the information, not just to officials but to the general public as well. Elected officials even use excerpts from SINERGIA's annual reports in their own public speeches. 39 The reports started in 2003 and are available electronically at: http://www.dnp.gov.co/PortalWeb/PND/InformealCongresodelaRep%C3%BAblica/tabid/210/Default.aspx. 18 2.4.2. Quarterly Reports and Monthly Online Performance Information Quarterly reports and electronically delivered monthly reports are a further innovation in disseminating information. The quarterly reports seek to make information available more quickly, so that feedback from specialized stakeholders, such as universities, research centers, or government technical staff interested in evaluation can be gathered. The monthly reports, meanwhile, seek to examine specific issues relating to government performance. Both types of reports also reflect a growing efficiency in the processing, administration, and consolidation of information owing to SIGOB, which has become a powerful information source with the largest consolidation of performance data on government policies and programs. Today, SIGOB is an efficient online accountability and information dissemination tool. It provides information in real time because as soon as administrative records or statistical information on performance indicators are produced, they are automatically loaded onto the system. An attempt was also made to broadcast radio programs on community stations in different parts of the country, to disseminate the most important national programs results to target audiences as well as to foster a discussion network. There were two series of these programs--one on social programs and the other on performance in the field of national security--both were delivered in clear and simple language that the general public could understand. The programs eventually ended because the DEPP resources needed to continue them were not available. 2.4.3. Town Hall Meetings and Accountability Councils of Ministers Two additional initiatives, outside the context of the reform but which have both benefited from and contributed to it, are the government's bi-weekly town hall meetings and the annual Accountability Councils of Ministers. The town hall meetings are a public control and accountability mechanism, which promotes interaction among national government, regional authorities, and the general public. Subjects of interest to each region are discussed at the meetings, and accountability is checked in terms of progress made on national government programs in the regions (see box 2). The Accountability Councils of Ministers, meanwhile, are held each year and are also led by the President. At these sessions, each minister presents his ministry's results to the country in a televised program and answers questions from the public. The televised councils are arranged by the President's Office, and SINERGIA/DEPP provides the necessary technical support for ensuring that the results information presented to the country is accurate and internally consistent. Neither the televised councils nor the town hall meetings set out to promote SINERGIA reforms, but they have clearly played a key role in reinforcing the results-based management reforms that SINERGIA started in 2002. 19 Box 2. Town Hall Meetings The town hall meetings have become a method for reinforcing Colombia's results orientation at the regional and local levels. The town hall meetings have been described as a national program to create national-local interaction and dialogue, focused on specific government issues of interest to the regions. Since August 2002, more than 203 town hall meetings have been held in different municipalities. They follow a results-based management methodology, which allows the national and local governments--ministers, governors, mayors, and agency managers--to set up commitments with the communities in the presence of the President. Commitments should be about priority social problems and be in line with the national and regional development plans. However, they cannot require budget resources other than what has already been approved in the annual budget by Congress. Therefore, commitments are mainly on solving operational problems and obstacles that limit the achievement of key goals for the regions as much as for the country. The general guidelines for the town hall meetings are set up in the NDP to increase transparency and efficiency, based on the following objectives: · Promoting citizen participation and public control through dialogue and direct interaction with national and local officials; · Stimulating permanent and effective accountability processes of the governments to the general public; · Stipulating coordination between different levels of government and public agencies, with the aim of increasing the government's effectiveness; and · Increasing transparency and citizen's confidence in public management. The town hall meetings take place in capital cities or key locations in the regions. The agenda is prepared ahead of the meetings by the President's Office, based on requests from regional or local governments. Most of the discussions follow the guidelines set out in the NDP's policies on education, social protection, security, etc. The mayors of the municipalities play a fundamental role in the town hall meetings. Monitoring of Town Hall Commitments and Completion of Tasks 2,346 2,500 2,000 1,500 1,030 1,087 1,000 500 229 0 Assigned tasks Completed In progress About to initiate tasks Source: Presidencia de la República ­ SIGOB, 2008 data. 2.4.4. Alliance with Civil Society Organizations Efforts made by SINERGIA to forge alliances with civil society organizations for stimulating demand of performance information are worth emphasizing. The aim was to ensure that these alliances became a mechanism for comparing and validating the results information produced by SINERGIA, as well as to provide an independent assessment of performance data and a channel through which to inform the general public about how government results came about. This is not a simple task, and it generated a series of difficulties for SINERGIA. Based on the experience of the successful Bogotá,Como Vamos? program40--an accountability and people's control initiative promoted by the private sector in the 40 Bogotá Como Vamos? (Bogotá, How Are We Doing?) monitors changes in people's quality of life, emphasizing compliance by the district administration to the National Development Plan. The project is an alliance between El Tiempo (the main national newspaper), the Corona Foundation, and the Chamber of 20 country's capital--SINERGIA suggested to the leaders of this program that a similar initiative should be launched at the national level. One of the key aims of this initiative was to stimulate demand for performance information from different sources, including SINERGIA's monitoring and evaluations. After a process lasting nearly two years, which involved significant work and technical agreements, the civil society representatives in the alliance decided to step down from their positions because they wanted to maintain an appearance of neutrality in the upcoming 2006 government elections, and the project fell apart as a result. A similar effort to exert public control on government results, and thus strengthening accountability, was made a year later by mobilizing the Colombia Lider initiative. This initiative was promoted by the business community, think tanks, and the media, and got under way in 2007.41 However, it remains to be seen whether it will accomplish its stated aim. 3. Success Factors and Obstacles A number of success factors as well as obstacles can be identified from the above discussion. 3.1. Role of the Department of National Planning as System Leader The DNP has been a champion in the development of evaluation in Colombia. The expertise of its staff has enabled it to produce M&E methods and instruments that are technically reliable. Similarly, its policymaking coordination role through CONPES, and the role it plays in preparing the government's investment budget, means that it can influence and help reduce resistance toward getting M&E practices incorporated into a whole-of-government system. The DNP advises the President and CONPES in determining what programs and funds to approve. It is involved in establishing macroeconomic policy and, together with the MoF, promotes fiscal stability. But unlike the MoF, which is concerned mainly with a balanced budget, DNP defends the investments needed to achieve each of the government's development goals. As far as the President is concerned, therefore, DNP is an effective counterweight to the MoF's power. For the sector ministries, DNP is sometimes a source of technical assistance and, at other times, a regulator when they stray from the planning and M&E frameworks.42 Having SINERGIA based in DNP has both advantages and disadvantages. The principal advantage is that it allows DNP technical units to consult and work reliably and easily with the ministries on drawing up investment programs. It therefore simplifies the task of integrating M&E into the national planning framework because it is easier to coordinate the definition of goals and indicators with ministries, as well as to carry out evaluations. The authority of the DNP, likewise, simplifies the task of giving advice on methodologies, best practices, and the development of evaluation instruments and capacities. Furthermore, apart from the President's Office, DNP is the only central agency that is Commerce. The monitoring is done in terms of access to quality goods and services, and takes public perceptions into account. See www.bogotacomovamos.org. 41 Colombia Lider (Colombia Leads) is a private initiative for monitoring and evaluating state results and government policies. It involves Semana Publications, the Colombian Banks Association (Asobancaria), the Antonio Restrepo Barco Foundation, the think tank Fedesarrollo, the United Nations Development Program, and the RCN radio and television channels. See www.colombialider.org. 42 This feature of Colombian institutions explains why a common opinion heard in certain technocratic circles is that without DNP, the President would be extremely dependent on the Minister of Finance's will. 21 interested in evaluating the government's overall results. Each ministry concentrates on its own sector and the MoF, which could perform this task, has neither the specific technical skills nor the interest and competencies in evaluation and sector policies. Placing responsibility for the national M&E System within DNP nevertheless brings with it certain disadvantages too, particularly, in terms of relations with ministries. Apart from what has already been said with respect to the MoF, the relationship with the sector ministries is, at best, contradictory. DNP is seen as an ally when it defends investment and provides technical assistance. However, when it points out problems with implementation and misalignment with the plans, it is viewed with mistrust. In recent years, as the ministries have acquired greater technical capabilities, it has to be acknowledged that this mistrust has grown. Regarding evaluation, however, the technical support provided by DNP is viewed as a net value added that no other entity is able to provide, at least until the sectors develop their own capacity in this area. 3.2. Constitutional and Legal Support The institutionalization of an M&E system is clearly a process that involves a cultural change as well as the development of capacities in the public sector. This takes time, and depends: (i) on the institutional and incentives framework on which it is based, and (ii) on the extent to which its products are used effectively. Experience has, in fact, shown that getting an M&E system incorporated in a sustainable manner throughout a government can take more than a decade.43 In the case of SINERGIA, constitutional and legal support for the system has been an important institutional factor in its success, for at least two reasons. First, there is a regulatory tradition in Colombia that prioritizes what is established in the legal framework, in order to ensure continuity from one administration to the next. Second, each administration's period in office is short (four years) and this--together with a tradition of high turnover, especially in the highest-level nominated positions--hinders evaluation institutionalization processes.44 SINERGIA's incorporation into Colombia's Constitution and laws has thus allowed the system to survive during times when support was low or there was lack of interest in M&E activities.45 Viewed another way, the legal mechanisms that exist have made it easier at critical times for SINERGIA to foster or maintain M&E practices in the government. There are nevertheless a number of disadvantages in the existing legal framework. In particular, the fact that it is linked exclusively to the planning system means that SINERGIA has no strong regulatory support for enforcing evaluation of the operating components of expenditure--these comprise 60 percent of the national budget and is the responsibility of the MoF. Although the latest amendment to the Budget Statute brought major progress in this area, procedural and regulatory interpretation problems remain the principal difficulty for integrating evaluation into the budget cycle. 43 This point is illustrated well by what happened in Australia, New Zealand, and Chile. See Mackay 1999 and 2007. 44 The ability to re-elect a President was introduced in 2006, making this aspect potentially less important in the future. 45 Special approval procedures must be followed in Congress in order to change the Constitution or a Statutory Law. 22 3.3. Role of the President's Office Measuring a government's performance is fundamentally political. The sitting President is accountable to voters and to Congress for the proposals made in election campaigns; and depending on how closely this principle is adhered to, Presidents will require information to measure government actions and to report to the general public on progress under their administrations. Since 2002, the President's Office has played a central role in the SINERGIA's institutionalization process, and is one of the main factors contributing to the system's success. The President periodically meets with ministers and directors to carry out high- level oversight of their performance, based on information from SINERGIA and the ministries, among other sources. Such high-level validation of the performance information helps to increase its use in government decision-making processes, and simplifies coordination at the technical level. In the words of President Uribe: "...spending ministries can use information from evaluations, basically, to administer its programs, to distribute their resources, to extend the programs, or to inform the development of their policies. However, if we ask ourselves what are the success factors in strengthening performance information in the country, the answer is a strong leadership and a firm will that is pressing the programs to obtain results, that is also pressing for good quality performance measurements, and for all government services to have performance indicators. The political pressure to improve performance is a factor of success."46 The active role of the President's Office brings SINERGIA both advantages and disadvantages. The advantages are that the M&E information it produces is used automatically by all government entities for managerial and accountability purposes. Reports to Congress, press releases, periodic reports, administrative controls, councils of ministers, and town hall meetings all extensively use data provided by the system. The disadvantages stem from mistrust in some sectors, and the fact that the President's Office is indirectly involved with evaluations when, in the opinions of many people, it has an obvious political interest in defending the government programs. The credibility of M&E information has been questioned on several occasions in public debates and in the media. It has also been suggested by critics that the government selectively presents the performance information. Even though there is no evidence to support such a charge, it can damage trust in the system and impede the system's full institutionalization. SINERGIA thus faces a dilemma. On the one hand, having the support of the President's Office has resulted in a real overall strengthening of the system, in the content and quality of the performance information, and particularly in its utilization by different sectors and audiences. On the other hand, such support also makes the system appear less independent, and M&E products could accordingly be trusted less. 3.4. Financing Strategy and Staffing From the start, SINERGIA has financed its activities with donor grant and loan funding, for two principal reasons: (i) because fiscal restrictions limited the chances of getting government funds for M&E activities, especially at the beginning, and (ii) because it has been clear to the government that this modality provides access to a continuous flow of international technical assistance, which has been a key element in the development of the 46 President Alvaro Uribe's opening remarks in the International Seminar on Fiscal Transparency; DNP- SINERGIA, February, 2006. 23 M&E system. Approximately US$2.2 million were invested between 1996 and 2001, an annual average of US$360,000 during the first stage.47 Initial funding was devoted toward meeting the system's design activities, improvements in methodology and information, implementation of pilot evaluations, training, publications, and payroll for DEPP professional staff. Allocations from the National Budget, about 27 percent, were only for equipment, operations facilities, and a small number of the DEPP staff (approximately 19 percent of the total staff needed). Due to the limited financing, the scale of evaluation activities did not extend beyond a small number of pilot initiatives.48 Moreover, DEPP's monitoring and evaluation team consisted mainly of consultants, with an extremely high turnover rate. A high attrition rate made it difficult to retain the skills sets developed and to keep a systematic record of progress made by the system. Staff salaries, when they are funded by loans and grants, are unstable from year to year. This situation is one of the main obstacles to the sustainability of SINERGIA activities. Despite various attempts to install a permanent technical staff, the problem still remains unresolved today. Though many analysts may see this problem as suggesting a rather weak commitment by DNP's top management to SINERGIA, it is fair to say that the creation of positions in the Colombian civil service has always been quite a cumbersome procedure and politically difficult to justify, particularly in times of fiscal constraint. Since the 2002 reform, the system has required a considerable increase in funding. In particular, more financing was needed to scale up M&E activities and to increase the number of qualified staff in advanced evaluation techniques. An additional finance component was included in a second phase of the Financial Management Modernization Program (PMAFP II) 49 funded by the World Bank. The sum allocated (US$1.5 million) was, however, insufficient to cover the cost of all the proposed M&E activities. The DEPP thus set into motion a financing strategy that had two objectives: (i) to make use of available funds and installed capacity as matching funds to bring in other sources of financing for SINERGIA activities, and (ii) to seek approval for recruiting 18 of the current system's consultants as permanent DEPP staff. In the first of these objectives, the system was extremely successful. More than US$13 million were raised and spent between 2002 and 2006, all coming from donor grants, loans, and government matching funds. During this period, the system's budget grew to an average of US$2.6 million per year--more than 7 times the annual amount that was spent during its first stage. However, it must be noted that the government's direct funding remained at 26 percent, a level that is still too low to ensure sustainability of the system if international funds were to divest (figure 4).50 47 Public Financial Management Modernization Program (PFAM I). World Bank. 48 It should be stressed that some agencies--especially in the education, social protection, and infrastructure sectors--financed other evaluations during the same period but these were undertaken outside the system. 49 Programa de Modernizacion de la Administracion Financiera Publica. 50 At the international level, financial support and support-in-kind was obtained from the Human Development Network (HDN), World Bank Institute (WBI), Independent Evaluation Group (IEG), and Poverty Reduction and Economic Management (PREM) in the World Bank; as well as from Social Sector, Institutional Development, and Knowledge (PRODEV) in the Inter-American Development Bank (IADB); and cooperation from the United States (USAID), The Netherlands, Germany (GTZ), and the International Monetary Fund (IMF). At the national level, sources included the Central Bank (Banco de la República), Acción Social (President's Office), the Ministry of Social Protection, the Ministry of Education, and the Ministry of the Environment and Housing. 24 In the absence of additional local funds and to ensure that these activities would continue until 2012, the DEPP got approval to prepare a new credit operation with the World Bank, initially estimated to be US$20 million. Implementation of this operation is to begin in 2009 but, due to fiscal restrictions, the amount was reduced to US$8.5 million by the new administration.51 As far as staff institutionalization is concerned, the aim of the strategy was not achieved. Despite obtaining initial approval to create the planned civil service positions, at an estimated annual cost of US$75,000, the decision was overturned in 2005. To help with the government's fiscal adjustment program, the state reform initiative had suggested a 20 percent cut in staffing by all national entities, and because it was impossible to reduce staff in other areas of the DNP, it became politically impossible to insist on the staffing initiative for SINERGIA/DEPP. Figure 4. SINERGIA Financing, 1996­2006 16.0 15.3 1996­2001* 2002­2006** 13.1 14.0 12.0 US$ Million 10.0 8.0 5.9 6.0 3.5 4.0 2.7 2.2 1.5 2.0 0.6 1.0 0.0 USAID&Others IDB Government Government World Bank World Bank Sub total TOTAL Sub total Sources of Funding * Figures are not available for the period 1991­1996. ** Includes US$100,000 in bilateral cooperation from the Netherlands. Does not include in-kind technical assistance from the German Agency for Technical Cooperation (GTZ) and the International Monetary Fund. Government resources correspond to agencies matching funds for undertaking evaluations. Source: SINERGIA. 3.5. Methodologies for Defining Indicators The process of defining indicators, goals, and baselines is crucial to the success of an M&E system. The quality of information and evaluations depends on this development, as does their relevance to decision-making processes. Also dependent on this process is the level of ownership and commitment to measuring results by those responsible for executing programs. Several SINERGIA success factors are listed here. 51 See CONPES 3515 of 2008. 25 The collaborative approach used to define and agree on indicators, in which executing entities, DNP sector experts, and SINERGIA/DEPP professionals are all involved. This enables the system to ensure that minimum quality standards are met in all sectors. The technical criteria used to set up indicators and goals, including: (i) target functions of each sector and agency, (ii) sources and frequencies of available information, (iii) programmed investment funding, and (iv) sector capacities for achieving the goals set. This gives goals a sense of reality and ensures measurability. The validation of indicators and targets by ministers and directors of each agency and the ratification of their commitment in an agreement letter sent to the President's Office and DNP. This step ensures that each minister and director owns its indicators and goals and that it is directly responsible to the President and the general public for the results. The extensive dissemination, in the form of printed charts and information on the SIGOB website. This enables the commitment to citizens to be expanded, exposes performance to scrutiny by the general public, and makes it difficult for entities to unilaterally lower their targets. 3.6. Availability and Quality of Performance Information Identifying strengths and weaknesses of performance information has been an important contribution made by SINERGIA since 2002. The system's whole-of-government approach has provided DNP/DEPP (its technical secretariat) with a unique perspective on the main information limitations, bottlenecks, and best practices in the country. As a result, DEPP strengthened its capacity to advise national and subnational governments and agencies on how to move toward a more systematic production of sufficient and good-quality performance information. For this reason, SINERGIA/DEPP was asked to prepare a national information strategy, as part of the Colombia's long-term plan: Vision 2019.52 In this role, SINERGIA helped to set out the Colombian policy guidelines and standards that national and subnational governments currently follow to further increase the quality and availability of administrative records and statistics, and to reduce their production costs. It also contributed to better institutional arrangements for improving coordination and harmonization of data collection and use at all levels, but particularly among line ministries, the National Statistics Agency (DANE),53 the MoF, the DNP, and the President's Office. Furthermore, it helped to define an action plan and specific guidelines for supporting the agenda of the National Inter-Sectoral Committee for Information Policy and Management (COINFO).54 It was hoped that a coherent and long-term information strategy would contribute to further institutionalization and consolidation of the evaluation system, and vice versa. 3.7. Inter-Sector Evaluation Committee Coordination mechanisms to identify performance information needs, to select methodologies, to choose the programs to be evaluated and, particularly, to ensure the use of evaluation findings have been effective in introducing a favorable climate for evaluation in Colombia. Such a climate has also enabled the number of evaluations to be increased and a community of evaluators to be built up in the country, though the latter is still in its early stages. 52 See DNP 2006. 53 Departamento Administrativo Nacional de Estadisticas. 54 Comision Inter-Sectorial de Politicas y Gestion de la Informacion. 26 The Inter-Sector Evaluation Committee, which was set up in 2002, has played a crucial role in these aspects. One specific development is that it has allowed better coordination processes to be implemented among SINERGIA/DEPP, MoF, DNP directorates, and the principal sector ministries, so that an evaluation agenda could be drawn up jointly (table 2). Another development is that the committee has also helped develop the evaluator market, especially in terms of impact methodologies. By encouraging competitive contracting processes, which favor technical quality and the forging of alliances between national and international experts, the number of evaluators from the main research centers and universities is growing. Table 2. Inter-Sector Evaluation Committee: Members and Functions Members Functions 1. Chaired by DNP Deputy Director 1. To define the SINERGIA conceptual and institutional framework. 2. President's Office ­ High Counselor 2. To propose programs to be evaluated. 3. Ministry of Finance ­ Budget 3. To draw up an annual and four-year evaluation agenda. Director (DGPN) 4. To identify sources of funds and provide support in obtaining 4. Ministry of Social Protection (MPS) funding to finance evaluations deemed to warrant priority. ­ Deputy Technical Minister 5. To approve methodologies that will be used in each evaluation. 5. Ministry of Education (MEN) ­ 6. To approve the contracting out of evaluations included on the Deputy Minister for Quality agenda. 6. Public Finance and Investment 7. To monitor implementation of evaluation findings. Director (DIFP) ­ DNP 8. To analyze recommendations made in evaluations and propose 7. Sector Directorates ­ DNP courses of action for making use of their results. 8. Guest agencies 9. Others, as the Committee may deem necessary for promoting the 9. SINERGIA/DEPP Technical implementation of the evaluations. Secretariat Source: DNP ­ DEPP. The Committee has encouraged the transfer and sharing of capacities and experiences among its member agencies and has stimulated the development of seminars, courses, and conferences. Several events were held during 2002­2006 to present the results of specific evaluations and to discuss the implications of their findings and need for developing evaluation capacities in the public sector, thereby helping to set up informal knowledge networks and stimulating academic interest in specific evaluation training programs.55 Thanks to this committee, the percentage of the nation's budget that has undergone some kind of evaluation increased more than tenfold between 2002 and 2006.568 Evidence also exists of influential decisions promoted by the committee and taken by the government, such as the Families in Action program (which had its budget increased and its coverage expanded), and the evaluation of the General Transfers to the Regions System. The DEPP has played an important part in helping to get these results by marking out the process, drawing up the agenda, coordinating actions resulting from the evaluations, suggesting programs to be evaluated, and periodically submitting concrete evaluation of program proposals to the committee for approval. Despite the boost that the Inter-Sector Evaluation Committee has managed to give to evaluation, there has also been a series of hindrances to its development. The more important of these are listed here. 55 Examples are the impact evaluation courses at Los Andes University and the Universidad de Antioquia, two of Colombia's most prestigious universities. 56 DNP and Presidencia de la Republica 2006. 27 The committee can advocate program changes in response to evaluation findings but it is an ad hoc group, without solid legal development or support, and therefore it does not have the power to enforce the implementation of changes.57 The coordinating body has no autonomy or budget of its own for promoting and financing evaluations in other areas. It depends on the funding made available by its members. All of its members are government officials; there are no external evaluators or academics and this reduces the perception of how independent and credible the coordinating body is. It can neither extend evaluations on a wider scale (e.g., to the regional level, or to other branches of the state) nor does it have any power to regulate the evaluators market. It depends heavily on the insufficient capacity of DEPP as the SINERGIA Technical Secretariat, which has limited staff to comply with a growing evaluation agenda. Despite these disadvantages, the mere fact that this ad hoc group exists--and continues to exist--speaks to a positive trend and shows the growing interest in, and institutionalization of, evaluation in Colombia. 3.8. Regional and Sector M&E Pilot Systems Another success factor was the change in SINERGIA's regional and sector approaches to developing M&E systems. This consisted of moving from a supply-based, global approach covering national, provincial and local governments to one driven by demand and based on pilot initiatives at the regional, sector, and agency levels. Two parallel pilot programs were promoted between 2004 and 2006 in the major cities of Medellín and Pasto. As a result of the enthusiasm and capacities of the enterprising mayors, these pilot projects managed to incorporate a strong results orientation into their respective administrations, using M&E tools for planning and budgeting.58 A joint initiative by the private sector and the major national media in 2007 recognized the mayors of these two cities as the best mayors in the country for their respective city's size. Although this recognition cannot be said to have been entirely due to these pilot programs, they were clearly among the main factors considered. It should be stressed that unlike the experience at the national level, these pilot programs achieved effective coordination in both planning and budgeting, despite the fact that the institutional architecture of these two levels is exactly the same as that in central government. The regional experience thus demonstrated that while the investment budget (part of the planning units) and operating budget (under the purview of the budget units) are functionally two separate entities, it need not constitute an insurmountable obstacle to implementing an M&E system and a results-orientated model. These pilot programs and their principal lessons have been analyzed at seminars and conferences, and have thus become widely disseminated. Accordingly, one of the most important aspects is that they set a good practice precedent for other regional administrations and for the central government itself.59 57 A valuable contrasting example is Chile's Ministry of Finance, which has unique powers in ensuring implementation of evaluation findings. 58 Medellín is the second largest city in Colombia, with approximately 3 million inhabitants, and is one of the most progressive. Pasto is a medium-size capital city with around 400,000 inhabitants. Another pilot program was introduced in June 2006 in the town of Tocancipá, which has a population of less than 100,000. 59 These initiatives were financed with international cooperation funding acquired by SINERGIA and with local funds, which were far greater than the outside contributions. 28 This model was also implemented at the sector level in the Ministry of Social Protection (MPS) as well as at the agency level in the Institute for Family Welfare (ICBF)60 and the National Roads Institute (INVIAS).61 Coupled with the national model implemented by the present government, the impact of these processes has acted as a stimulus for a strong results orientation in other regional administrations, sectors, and agencies, which are now implementing similar approaches to results-based budgeting and management.62 The ability of SINERGIA/DEPP to meet the demands of municipal and provincial governments, as well as of other sectors and agencies for technical assistance is nevertheless limited. The model implemented initially envisioned technical assistance being provided by private entities and/or universities, under an arrangement regulated by SINERGIA. However, few entities with the capacity to provide the proper technical assistance and advisory services have emerged, thus limiting the potential for developing this model on a larger scale. 4. Recent Developments State reform has been relegated to lower priority in President Álvaro Uribe's second term, and other issues, notably internal politics and national security, have overtaken the government's attention. Furthermore, SINERGIA/DEPP and budget directors at DNP and MoF stepped down at the end of the first administrative term, and new personnel now occupy the top positions in both entities. A new institutional context, with notably less attention placed on evaluation matters, has once again made the system less dynamic. Although the reforms that were introduced during the first Uribe administration resulted in the general public favourable approval of M&E functions, some groups in academia and the private sector have increasingly questioned how independent SINERGIA really is, and how credible is the information that it provides. Critics argue that as long as SINERGIA remains under the control of the executive, there can be no guarantees as to how objective its evaluations are.63 Added to this is the loss of dynamism that the Inter-Sector Evaluation Committee has begun to show, due to its lack of autonomy, funds, capabilities, and powers to regulate the evaluation market, irrespective of how interested government officials are or are not in the subject.64 In the context of a reform that implemented major changes in the public sector's orientation toward results, this kind of scrutiny of the government by outside agents should also be seen as an indicator of the extent to which SINERGIA's products are used and valued, as well as an indicator of the progress made toward institutionalizing M&E in Colombia. The danger is that such developments could foster the conditions for a new period of stagnation and decline of the M&E system in the near future. Despite this current state, a new credit operation with the World Bank is expected to inject fresh funds into SINERGIA in 2009. Apart from giving specific M&E actions at national and local levels a boost, this program will concentrate most of its efforts on reinforcing the quality, reliability, and credibility of M&E information.65 The encouragement of external audits of SIGOB information and the setting-up of independent academic observation, 60 Instituto Colombiano de Bienestar Familiar. 61 Instituto Nacional de Vias. 62 Examples of these are cities like Barrancabermeja and Bucaramanga, or provinces like Cundinamarca and Cesar. The Ministries of Education, and International Commerce are additional examples. 63 See ANIF 2007 and 2008. 64 The Committee has reduced its role in bringing the supply of and demand for evaluations in the government into line, in ensuring methodological quality of evaluations, and in channelling evaluation findings toward decision-making processes. 65 World Bank. Monitoring, Evaluation, and Information Project. April 2008. 29 analysis and monitoring bodies for evaluations and the results thereof, including evaluations of the NDP, could make an important contribution toward enhancing the system's credibility. Using these funds as intended in a strategic, opportune manner could also help to create a new environment in favor of evaluations in the government, especially if they are conducted by credible evaluators and the results provide information that is relevant to the MoF, Congress, the control bodies, and the general public. However, all these factors could be of a purely immediate nature. In the coming years, if there is a real desire to consolidate the system and a results orientation in the public sector, a more far-reaching institutional re-design will be needed. This would ideally, once and for all, settle the issues of lack of authority and independence in the system, as well as problems associated with low levels of government funding, lack of regular staffing, and coordination and integration of M&E activities within the government. The examples of Mexico and Spain--where National Evaluation Commissions have been set up--show the extent to which institutionalizing evaluation as a regular state practice, with high-quality and objective standards, requires a guiding body with greater independence. Care should nevertheless be taken not to overestimate the value of this autonomy because involvement by key government bodies will always be fundamental. Whereas full independence reflects an ex-post control model, which is closer to that of supervisory organizations, performance monitoring and evaluation that uses a results-based approach is inextricably linked to the possibilities of providing a functioning government with active feedback. This is one of the main strengths of SINERGIA. Various alternatives should thus be examined, but the focus of the reform will have to affect at least five key levels separate from SINERGIA/DEPP: ministries and execution entities, DNP and its other directorates, the President's Office, the Inter-Sector Evaluation Committee, and CONPES. An institutional change of this kind will, of course, mean that further legal modifications will have to be made, and the main challenge will lie in getting support from within the government and also from sectors of Congress and academia. The progress that has been achieved since 1994 has nevertheless established a number of basic principles that could help to make the change easier. An important "culture" of measuring results has been established in government, and an effective practice of accountability to Congress is also in place. Likewise, though the basic M&E infrastructure could be improved, it at least exists, and it is used by the central administration. Furthermore, as a result of the system's promotion, evaluation skills have improved in academia and research institutions. Taken together, these are important assets that could have a favorable effect on the speed and scope of reform. 5. Lessons Learned and Conclusion Many lessons for other countries can be drawn from SINERGIA's experience, but the need to adapt them to the specific political, institutional, and cultural context of each country should not be overlooked. Some of the lessons learned are described here. Setting up and consolidating a whole-of-government M&E system is not just a technical or policy initiative because, in actuality, it is an ongoing public sector reform, which involves all administrative areas, sets out to change public officials' behaviors, and has to be in line with the broadest political objectives (e.g., democratization, governance, accountability, etc.). 30 Given the long-term effort and change in "culture" that a national M&E system entails, regulations can provide a firm basis for ensuring that the system will endure through changes in government administrations. Such regulations need to decrease or circumvent resistance from defenders of the status quo, while protecting the progress already achieved. The principal assets that a system boasts are a technically sound design, based on a diagnosis of the initial results' framework, and robust methodologies for drawing up indicators, goals, and baselines for monitoring and evaluating performance. Information quality, and hence credibility, depend on these things. Achieving them requires skills and an ability to combine local and international experiences. Identifying indicators based on variables that are relevant to the President, citizens, and decision makers is a central aspect, so that a balance can be guaranteed between the supply and demand of performance information. Simplifying the indicators base and concentrating it on the upper end of the implementation chain (outputs, outcomes, and impacts) is a prerequisite for this balance. Computer systems should be viewed as an M&E system tool rather than as the M&E system itself. These information instruments should be adapted to system needs, rather than M&E systems being adapted to the tools. A coordinating body with a strong technical capability and influence is a necessary condition, but is not enough by itself to ensure that progress will be made with institutionalization. Bringing numerous players and agendas into line, reacting to changing circumstances, and taking advantage of new opportunities to develop the system require management, a sense of opportunity, and strategic vision, not to mention support from entities with sufficient rank and political power, such as the President's Office or Congress. If continuity is to be achieved during the process of implementing a national system, an evaluation policy or strategy with clear objectives and well-defined components and incentives is necessary. Also required is constant dissemination of the policy's content, both within and outside the government. Although problems can be caused by strong planning and budget frameworks existing side-by-side, with investment and operation budgets functionally separate, such problems are not insurmountable obstacles to implementing an M&E system. This has been shown by the experiences gained from the pilot programs in Medellin and Pasto. Basic steps like defining program classifications of the budget and results-based budget presentational exercises can be influential when it comes to promoting a greater results orientation in expenditure. Such steps are important because they establish foundations for future and more complex results-based management endeavors. The development of evaluation capacities is a prerequisite. Intervention of M&E authorities is needed to increase the supply of qualified evaluators, either directly (through training programs) or indirectly (through contracting incentives). The demand for M&E information is the main driving force behind a system. Such information, however, does not occur on its own. Demand should be stimulated by generating a supply of usable performance information and ensuring that it reaches key users. The market for M&E information is imperfect, and public intervention is therefore needed. The credibility of a system is built around the quality of its M&E products, and not necessarily on its independence. The latter is necessary in advanced stages of the consolidation process, but the fact that feedback on administration requires active participation by government agencies should be taken into account. 31 Re-valuing the importance of accountability to Congress by producing annual results reports, based on M&E information, requires the use of simple formats and language, as well as communication and dissemination strategies. Establishing pilot M&E programs is an effective strategy for reducing the risks of progressing too quickly (e.g., inability to assimilate) or minimizing the costs of too slow an implementation (e.g., loss of interest). Although the budget model is the most widely promoted in public management, it is not suitable in all cases. Colombia offers the example of an alternative model, one that is planning-driven. For the past 15 years, SINERGIA has been the main reference point for Colombia's reform to a performance-based administration. The system has endured many changes to regulations and official policy by various administrations, and many of its tools have increasingly become the accepted practice in the public sector, especially in the central government. Most efforts have been aimed at introducing M&E practices into the planning and budgeting processes, especially in central government. However, the results-based planning model has superseded the budgeting model, owing mainly to the architecture of the central administration and the fact that the system is institutionally based in the DNP. As compared with similar experiences in other countries, SINERGIA's institutionalization process is notable for the way it has combined high-ranking, wide-ranging formal mechanisms (e.g., constitutional mandate and laws covering the whole of public administration) with the development of informal practices in key areas of the public sector (e.g., M&E culture in ministries and entities). Using lessons learned from international experience, Colombian's DNP has followed an implementation strategy, which--in line with the political and institutional reforms that have been going on in the country--brings together a uniform, progressive approach for the national administration with the operation of selective, demand- based pilot programs at regional and sector levels. The analysis presented in this paper has shown some of the steps taken by Colombia to implement a national M&E system since 1991, highlighting some of the major achievements and obstacles encountered. SINERGIA can be categorized as a whole-of-government M&E system. Its design, implementation, and reform have been based on best international practices and theories, with significant innovation and self-development components incorporated as it has evolved. Significant effort has been made in the country and implementation of M&E can be considered a success in many aspects. However, much more work needs to be done, and more creativity will be required if Colombia is to fully institutionalize M&E at all levels of government. Current circumstances indicate that the system may be entering a new deceleration phase. If Colombia truly wants to capitalize on what it has gained--given the alternating periods of fast progress and stagnation in which it has evolved--it must provide greater consistency and a broader institutional dimension to reform. What is needed is a system that depends less on the extent of current high-level support, is more autonomous and highly ranked, and is based on more powerful incentives. Such a system, however, will entail an altogether different institutional architecture, which may require Colombia to wait until the next phase of major state reform. 32 Bibliography Acosta, O. L., y Ayala, U. 2000. "Análisis del proceso presupuestal colombiano," Fedesarrollo, Mimeo, Bogotá, DC. ANIF (Asociacion Nacional de Instituciones Financieras). 2007. Comentario Economico. "Evaluación del Plan Nacional de Desarrollo 2002­2006: Resultados y Credibilidad." Octubre. Bogota, DC. ------. 2008. Comentario Economico. "Evaluación de Políticas Públicas por Objetivos: ¿Auto- Evaluaciones?," Agosto. Bogota, DC. Banco Mundial ­ Centro Latinoamericano de Administracion para el Desarrollo. 2007. "Colombia y el Sistema de Monitoreo y Evaluacion ­ SINERGIA," en Fortalecimiento de los Sistemas de M&E en América Latina. Bogotá, DC. Cárdenas, M., Junguito R. y Pachón M. 2005. "Political Institutions and Policy Outcomes in Colombia: The Effects of the 1991 Constitution," Fedesarrollo y BID. Bogotá, DC. Colombia. 1994. Ley orgánica del Plan Nacional de Desarrollo. Ley 152 de 1994, Bogotá D.C. Consejo Nacional de Política Económica y Social (CONPES). 1995. "Gestión pública orientada a resultados," Documento CONPES 2790. Bogotá, DC. ------. 1997. "El Salto Social, Informe de avance del Plan Nacional de Desarrollo," Documento CONPES 2917. Bogotá, DC. ------. 1999. "Acuerdos de Eficiencia: Estrategia de Evaluación del Plan Nacional de Desarrollo Cambio para Construir la Paz," Documento CONPES 3048. Bogotá, DC ------. 2001. "Gestión pública orientada a resultados. Compromisos sectoriales 2001," Documento CONPES 3106. Bogotá, DC. ------. 2002. "Evaluación de Impacto de Programas Sociales," Documento CONPES 3188, Bogotá, DC. ------. 2003. "Renovación de la Administración Pública," Documento CONPES 3248, Bogotá, DC. ------. 2004. "Renovación de la Administración Pública: Gestión por Resultados y Reforma del Sistema Nacional de Evaluación," Documento CONPES 3294, Bogotá, DC. Corral, L.C. 2006. "Evaluaciones Focalizadas en el Marco del Sistema Nacional de Evaluación de Resultados," Mimeo, SINERGIA, Departamento Nacional de Planeación, Bogotá, DC. Currie, L. 1984. "Una nota sobre los planes nacionales y modelos" en Revista Desarrollo y Sociedad, No. 14. Universidad de los Andes. Bogotá, DC. Curristine, T. 2006. "Performance Information in the Budget Process: A Comparison of the OECD Countries." In Fiscal Transparency. Proceedings of a SINERGIA/World Bank/Inter- American Development Bank and International Monetary Fund conference. Bogotá, DC. Departamento Nacional de Planeación (DNP). 1994. "Control y evaluación de la gestión pública. Los desafíos de la nueva Constitución," Bogotá, DC. ------. 1996. "Hacia una gestión pública orientada por Resultados." Lito Camargo, Bogotá, DC. ------. 2003a. Plan Nacional de Desarrollo 2002­2006: Hacia un Estado Comunitario. Bogotá, DC. ------. 2003b. Metas del Plan Nacional de Desarrollo: Un Compromiso del gobierno Uribe. SINERGIA. Bogotá, DC. ------. 2005. Instrumentos para la Ejecución, Seguimiento y Evaluación del Plan de Desarrollo Municipal: Plan Indicativo, Plan Operativo Anual de Inversiones y Plan de Acción. Corporacion Andina de Fomento ­ Direccion de Desarrollo Territorial Sostenible. ------. 2006. "Visión Colombia II Centenario: 2019," Bogotá, DC. Departamento Nacional de Planeación (DNP) y Presidencia de la Republica. 2003. Informe del Presidente de la Republica. Alvaro Uribe Velez al Congreso. Bogotá, DC. 33 ------. 2004a, 2005a, y 2006. Informe del Presidente de la Republica. Alvaro Uribe Velez al Congreso. Bogotá, DC. ------. 2004b y 2005b. Balance de Resultados y Metas del Ano y del Cuatrienio. SINERGIA. Bogotá, DC. Departamento Nacional de Planeación y Alta Consejería Presidencial de la Presidencia de la Republica. 2004, 2005, y 2006. "Presupuesto de Inversión por Resultados." SINERGIA. Bogotá DC. Departamento Nacional de Planeación y SINERGIA. 2004. "Programa Familias en Acción: condiciones iniciales de los beneficiarios e impactos preliminares." Serie Evaluación de Políticas Públicas. Escobar, S. 1996. "El Sistema Nacional de Evaluación de Resultados", en Hacia una Gestion Publica Orientada a Resultados, Departamento Nacional de Planeación. Bogotá, DC. Fernández, A. M. y González, J.D. 2006. "El Seguimiento a las Políticas Públicas en Colombia", Mimeo. SINERGIA, Departamento Nacional de Planeación, Bogotá, DC. Gandour, M. 1996. "La Evaluación de Resultados en el Contexto de un Proceso de Democratización," en Hacia una Gestión Publica Orientada a Resultados. Departamento Nacional de Planeación, Bogotá, DC. Guerrero, R. P. 1999. "Comparative Insights from Colombia, China and Indonesia". Evaluation Capacity Development Working Paper Series No. 5, World Bank, Washington DC. International Monetary Fund. 2005. Colombia Public Investment and Fiscal Policy. Fiscal Affairs Department. April. Washington, DC. Mackay, K. 1999 "Fortalecimiento de la capacidad de evaluación. Guía para el diagnóstico y marco para la acción." Documento N.6. Departamento de evaluación de operaciones del Banco Mundial. Washington, DC. ------. 2007. "A Diagnosis of Colombia's National M&E System ­ SINERGIA." Evaluation Capacity Development Working Paper No. 17, Independent Evaluation Group, World Bank, Washington DC. Ospina, S. y Ochoa, D. 2003 "El Sistema Nacional de Evaluación de Resultados de la Gestión Publica (SINERGIA) de Colombia," en Evaluación de Resultados para una Gestión Publica Moderna y Democrática: Experiencias Latinoamericanas. Caracas: Centro Latinoamericano de Administracion para el Desarrollo (CLAD). SINERGIA. 2004. "Programa Familias en Acción: condiciones iniciales de los beneficiarios e impactos preliminares." Evaluación de Políticas Públicas No. 1, DNP, Bogotá, DC. Vélez Uribe, Álvaro. 2002. "Firm Hand, Big Heart. The Path towards Recovering Trust." May. Villarreal, Julio. 2007. "El Sistema de Evaluacion de Colombia" en Evaluación de Resultados para una Gestión Publica Moderna y Democrática: Experiencias Latinoamericanas. Caracas: Centro Latinoamericano de Administracion para el Desarrollo (CLAD). Wiesner, E. 2000. "La evaluación de resultados en la modernización del Estado en América Latina. Restricciones y estrategias para su desarrollo". En: Archivos de Macroeconomía ­ DNP. Documento 136. Abril. http://www.dnp.gov.co. World Bank. 1997. Colombia: Paving the Way for a Results-Oriented Public Sector. Washington, DC. ------. 2005. Colombia Public Expenditure Review. Poverty Reduction and Economic Management Unit. April. Washington, DC. World Bank and IDB (Inter-American Development Bank). 2006. "Towards the Institutionalization of Monitoring and Evaluation Systems in Latin America and the Carribean." Proceedings of a joint conference. Ed. by Ernesto May and others. Washington, DC. Zaltsman, A. 2006. Experiencia con la Institucionalización de Sistemas de Monitoreo y Evaluación en Cinco Países Latinoamericanos: Argentina, Chile, Colombia, Costa Rica y Uruguay. Desarrollo de la Capacidad de Evaluación. No. 16. Banco Mundial. 34 Annex 1. Evolution of SINERGIA's Legal Framework Legal Mandate Purpose Political Constitution of 1991, Defines responsibility of DNP for designing and organizing Article 343 public administration results evaluation systems. Restructures DNP and sets up the Special Administration Decree 2167 of 1992 Evaluation and Control Division as responsible for designing the administration evaluation and control system. Establishes the general objectives, structure, and strategy for CONPES Document 2688 of implementing the National Administration Results and 1994 Evaluation System. Statutory Law which establishes the obligation on DNP to Law 152 of 1994 carry out follow-up and evaluation and control of development plans. Conceptualizes results-orientated public administration and CONPES Document 2790 of establishes the Estimated Plan as the instrument for 1995 evaluating development plan results. Upgrades SINERGIA technical unit from division to Decree 1363 of 2000 directorate level in the DNP. CONPES Document 3248 of Establishes the Public Administration Renewal Program, 2002 including administration by results as a cross-cutting reform. Sets up the 2002­2006 National Development Plan, including Law 812 of 2003 guidelines and provisions for results-based management and evaluation of results. Restructures DNP and strengthens SINERGIA technical Decree 195 of 2004 functions by creating the Public Policy Evaluation Directorate with extended sphere of action. Comprehensively reforms SINERGIA, amends its scope, CONPES Document 3294 of institutional framework, and structure. Forms a top-ranking 2004 political backup for institutionalizing results-based M&E in Colombia. Modifies the Statutory Budget Law by including spending program classifiers, evaluation of expenditure as a key part of Decree 4730 of 2005 the budget cycle, and impact evaluation of main budgeted programs, as defined by DNP and the Ministry of Finance in the SINERGIA evaluation agenda. Four year mandate that created the Inter Sector Evaluation Law 1151 of 2007 Committee as a coordination mechanism to harmonize supply and demand of evaluation, select methodologies and stimulate utilization of findings. 35 Annex 2. SINERGIA Evaluations, 2002­2008 Percentage of total Evaluations Completed Cost (US$) evaluation budget (%) Cash Transfers - Families in Action rural 2,000,000 19.84 Social Housing (VIS) 180,430 1.79 Training - Youngsters in Action 700,000 6.94 Colombian Micro, Small, and Medium Enterprise Modernization and Development Fund (FOMIPYME) 86,957 0.86 Public Administration Renewal Program (PARP) 25,000 0.25 - Public Procurement Reform Program Public Administration Renewal Program (PARP) 111,957 1.11 - Enhancing Productivity of Prioritized Agencies (1st Phase) Direct Employment Support Program (DESP) 30,000 0.30 Workfare - Jobs in Action 1,500,000 14.88 Food Security Network (FSN) 300,000 2.98 Peace and Development Laboratories Program (1st Phase) 237,113 2.35 Subtotal 5,171,457 51.29 Percentage of total Evaluations in Progress Cost (US$) evaluation budget (%) Cash transfers - Families in Action (Internally Displaced Population) 278,261 2.76 Cash transfers - Families in Action (Urban Pilot) 93,485 0.93 Cash transfers - Families in Action (Urban Expansion) 386,957 3.84 Nutritional and Child Care Program - Community Homes 1,347,826 13.37 General Subnational Transfers System 1,086,957 10.78 Socioeconomic Stratification System 201,739 2.00 Public Administration Renewal Program (PARP) 25,000 0.25 - Legal Management reform Public Administration Renewal Program (PARP) 25,000 0.25 - Public Assets Administration reform Public Administration Renewal Program (PARP) 25,217 0.25 - Public Service Reform Public Administration Renewal Program (PARP) 15,652 0.16 - Enhancing Productivity of Prioritized Agencies (2nd phase) Peace Work - Acción Social (Executive) n.a. Disabled Persons Program - DPP (Executive) 28,783 0.29 Ministry of the Interior and Ministry of Justice merger (Executive) 10,435 0.10 Rural Development - Agrisector Safe Income Program 1,234,338 12.24 Strengthening Culture - Batuta 108,660 1.08 Modernization of the Attorney General Program 21,739 0.22 Urban Integral Plan 20,909 0.21 Subtotal 4,910,958 48.71 TOTAL 10,082,415 100 Source: Department of National Planning / DEPP. Note: n.a. = not available. 36 Other Titles in the ECD Working Paper Series #1: Lessons from National Experience, by Keith Mackay, 1998. #2: Zimbabwe: Issues and Opportunities, by Stephen Brushett, 1998. #3: Indonesia's National Evaluation System, by Alain Barberie, 1998. #4: The Development of Australia's Evaluation System, by Keith Mackay, 1998. #5: Comparative Insights from Colombia, China and Indonesia, by R. Pablo Guerrero O, 1999. #6: Evaluation Capacity Development: A Diagnostic Guide and Action Framework, by Keith Mackay, 1999. #7: Sub-Saharan Africa: Lessons from Experience in Supporting Sound Governance, by Mark Schacter, 2000. #8: Strengthening Capacity for Monitoring and Evaluation in Uganda: A Results Based Management Perspective, by Arild Hauge, 2001. #9: Guide to Conducting Reviews of Organizations Supplying M&E Training, by Marie-Hélène Adrien, 2003. #10: The Development of Monitoring and Evaluation Capacities to Improve Government Performance in Uganda, by Arild Hauge, 2003. #11: Two Generations of Performance Evaluation and Management System in Australia, by Keith Mackay, 2004. #12: An Assessment of the Impact of Bangalore Citizen Report Cards on the Performance of Public Agencies, by Adikeshavalu Ravindra, 2004. #13: Building Country Capacity for Monitoring and Evaluation in the Public Sector: Selected Lessons of International Experience, by Salvatore Schiavo-Campo, 2005. #14: Evaluation Capacity Development in the Republic of Ireland, by Richard Boyle, 2005. #15: Institutionalization of Monitoring and Evaluation Systems to Improve Public Sector Management, by Keith Mackay, 2006. #16: Experience with Institutionalizing Monitoring and Evaluation Systems In Five Latin American Countries: Argentina, Chile, Colombia, Costa Rica and Uruguay, by Ariel Zaltsman, 2006. #17: A Diagnosis of Colombia's National M&E System, SINERGIA, by Keith MacKay, 2007. 37 Other Recommended Reading Annual Report on Evaluation Capacity Development. OED, 2002. Conducting Quality Impact Evaluations Under Budget, Time, and Data Constraints. IEG, 2006. Developing African Capacity for Monitoring and Evaluation. Development Bank of Southern Africa, African Development Bank, and World Bank, 2000. Evaluation Capacity Development: OED Self-Evaluation. OED, 2004. Evaluation Capacity in Africa: Selected Proceedings from a Seminar in Abidjan. African Development Bank and World Bank, 1998. Evaluation Capacity in Asia: Selected Proceedings from the International Conference, Beijing, 1999. United Nations Development Programme, Chinese National Center for Science and Technology Evaluation, and World Bank, 2000. Glossary of Key Terms in Evaluation and Results Based Management. Development Assistance Committee, OECD, 2002. How to Build M&E Systems to Support Better Government, K. MacKay, 2007. Impact Evaluation The Experience of the Independent Evaluation Group of the World Bank. IEG, 2006. Influential Evaluations: Detailed Case Studies. OED, 2005. Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs. OED, 2004. Monitoring and Evaluation: Some Tools, Methods and Approaches, 2nd Edition. OED. 2004. OED: The First 30 Years. P. Grasso, S. Westy, and R. Weaving (eds.), OED, 2003. Public Sector Performance--the Critical Role of Evaluation. K. Mackay (ed.), OED, 1998. The Role of Civil Society in Assessing Public Sector Performance in Ghana. K. Mackay and S. Gariba (eds.) 2000. Towards the Institutionalization of Monitoring and Evaluation Systems in Latin America and the Caribbean: Proceedings of a World Bank / Inter-American Development Bank Conference. E. May, D. Shand, K. Mackay, F. Rojas, and J. Saavedra (eds.), 2006. ____________________________________________________________ Other relevant publications can be downloaded from the ECD website at http://www.worldbank.org/ieg/ecd. 38