34642 INTERNATIONALBANK FOR WORLD BANK R E T C N O E N STRUCTION PM AND DEVELO September 2005 No. 78 A regular series of notes highlighting recent lessons emerging from the operational and analytical program of the World Bank`s Latin America and Caribbean Region INSTITUTIONALIZING MONITORING & EVALUATION SYSTEMS: FIVE EXPERIENCES FROM LATIN AMERICA Ruxandra Burdescu, Azul del Villar, Keith Mackay, Fernando Rojas, Jaime Saavedra Background The Chilean Experience1 Many governments in the Latin America and Caribbean The Chilean system of management controls and results- region (LAC) have gained an increased understanding of based budgeting consists of the following tools: perfor- the value of M&E to help both governments and donors mance indicators; program and agency evaluations (in- alike better understand what public investments and cluding Comprehensive Spending Reviews); a Bidding interventions work well, which do not, and the reasons Fund for public programs; Management Improvement why. Monitoring the performance of public programs Programs linked to performance bonuses for central and institutions helps increase their effectiveness, pro- government employees; and Comprehensive Manage- vides increased accountability and transparency in how ment Reports. These tools are all integrated into the public monies are used, and can inform the budgetary budget process, and create synergies from the concep- process and the allocation of public resources, thus tual elements in their design and implementation. Care improving their effectiveness to improve welfare and, has been taken to maintain consistency in the methodol- consequently, reduce poverty and increase the equality of ogy, and to emphasize feedback.2 opportunities. To further promote the institutionalization of M&E "Chile's system is one of the strongest govern- systems, the World Bank, in partnership with the Inter- ment M&E systems in the world. In achieving American Development Bank (IDB), organized a re- this success, the single most important success gional conference on June 6-7, 2005 to take stock of the factor is probably the location of the M&E lessons learned. The participants represented finance system - where the system was designed, pro- and sector ministries from eleven countries, as well as gressively developed, and is still managed - in experts from academia, bilateral donor agencies, consult- the Ministerio de Hacienda [Ministry of Fi- ing firms, and various sectors of the host institutions. nance], specifically its Budget Directorate Experiences from five countries, namely Chile, Colom- (DIPRES). Hacienda has succeeded in creat- bia, Mexico, Brazil and Peru, were shared during the ing an "evaluation factory", which includes a conference, and served as input to determine lessons well-developed process for planning, commis- learned in institutionalizing systems of monitoring and sioning, managing, reporting and using a evaluation. The conference also served as a springboard range of types of evaluation." to launch an informal network of policymakers and M&E practitioners to allow such national experiences to be -Keith Mackay, Senior Evaluation Officer, Op- shared much more widely and systematically throughout erations Evaluation Department, The World Bank the LAC region. 1 and Results-Based Management was created in 2002 Other notable features of Chile's M&E system include specifically to respond to this concern. Indicator over- that: load has also resulted in efforts to simplify performance · DIPRES contracts out all evaluations to academia and indicators, limiting their number and enhancing their consulting firms. The Ministry of Finance manages this specificity. This process of simplification will assist process very well, using standardized terms of reference officials in analyzing the relationship between policies/ for the evaluations, an open and transparent process of priorities and expenditures. Finally, the M&E system has selecting consultants, and making all evaluation reports been concentrated at the central level thus far. But publicly available via the DIPRES website. The reliance replication of the model is planned at the sub-national on external evaluators, managed by a central ministry, level, and pilot programs are already underway for provides a model for other governments to consider. The approach encourages the independence and objectivity municipal governments. of evaluators. AWorld Bank study3 found the evaluations to have a high level of credibility among key stakeholders M&E of Social Programs in Mexico in the executive and legislative arms of the Government, and in academia. · Evaluation results are strongly integrated into decision The monitoring and evaluation of social programs in making and impact the budget via confirmation of pro- Mexico is administered through the National Council of gram effectiveness, major program redesign, or even Evaluation of Social Policy. A series of rigorous impact program abolition. The World Bank review found that the evaluations of large social programs, such as the Ministry of Finance also uses the evaluation findings Progresa/Oportunidades program, have strengthened the intensively in other ways, including changes in program process of M&E in the country.4 These evaluations have management that it requires sector ministries to imple- proved highly influential, by persuading the government ment, and in its budget reports to the Congress. Indeed, to retain these programs and to expand their scope the Ministry of Finance collects regular information on considerably. Congress also has assumed an active role the extent and manner in which the evaluation findings in the process, mandating program evaluation of social are used. · programs, as well as auditing reports of performance Intensive training is provided to the evaluation commu- indicators. nity. Colombia's SINERGIA Since 1994, the National System for Evaluation of Management and Results (SINERGIA for its acronym in Spanish), has been in operation to track and measure public sector performance. In particular, the government seeks to use the results obtained through the M&E system to improve resource allocation, assist in the formulation of the National Development Plan, and provide informa- tion for debate on public policies. SINERGIA consists of three principal components: 1) results monitoring, 2) strategic evaluations, and 3) report- ing for accountability or social control. The system also includes a web-based information tool, the System of Programming and Management by Objectives and Results (SIGOB) that provides perfor- mance information in real time. Given the strong Mexico has faced similar challenges as other countries in emphasis on accountability to civil society, the website the region in the implementation of its M&E system. The serves as an effective resource to disseminate informa- extensive process of developing and maintaining perfor- tion to the public. mance indicators needs further simplification. Not only that, the linkages between evaluation results and budget- There is a need to further strengthen the relationship ary decision-making require strengthening in order to between evaluation results, and planning and budget achieve a more systemic influence. The national M&E preparation. An Inter-Sectoral Committee for Evaluation system is also seeking to identify synergies with evalua- 2 tion initiatives at the state and municipal levels to expand are linked to strategic products. Indicators are also being its reach. The Ministry of Finance and the sector developed for the sub-national government, where much ministries are in the process of developing a stronger of the responsibility for basic service delivery lies. Yet as framework for a results-based M&E system. the system is still in the initial stages of development, an evaluation framework, including evaluation tools and The Plan Plurianual and M&E in Brazil criteria to define which programs will be evaluated, remains to be determined. The centerpiece of the Brazilian system is the national Lessons Learned Plan Plurianual (PPA), a four-year development agenda that is enshrined in the Constitution. The PPA serves as a key vehicle for addressing strategic govern- It was evident from the country experiences presented ment programs. The M&E system influences the PPA that there is no single "destination" for countries in terms by linking the evaluation results with the development of what a well-performing M&E system looks like. Some plan, and it endeavors to support the creation of an countries stress a system of performance indicators, evaluation culture to enhance policy formulation and while others focus on carrying out evaluations (program program implementation and management. The M&E reviews or rigorous impact evaluations). And while some system focuses on assessing program objectives and countries have created a whole-of-government approach performance across organizational boundaries, and cov- driven by finance or planning ministries, others are more ers all public expenditures, integrating capital and current focused on sector M&E systems. One key characteris- accounts. At the same time, few programs have tic of most of the systems that are now at different stages adequate performance indicators. of implementation in LAC, is that they reflect country- led, rather than donor-driven, efforts to institutionalize However, the links between evaluation results and the M&E. development of the executive budget are not clear, and even less so with the Congressional budget, although the law requires that the annual budgets reflect the priorities defined by the PPA. A centralized information sys- tem, known as SIGPlan, disseminates data and information from the Minis- try of Finance to the public. And efforts are underway to integrate the sector ministries' information systems (INFRASIG) with SIGPlan to provide a more unified source of information to facilitate the use of M&E in public decision-making. M&E of Public Expenditures in Peru Yet the shared experience of these countries has led to The Ministry of Economy and Finance of Peru launched some collective wisdom about the development of solid the National System of Monitoring and Evaluation of M&E systems: Public Expenditures (SSEGP) in 2004 as part of a wider · There is a need to clearly define the roles and responsibili- effort to promote results-based management. SSEGP ties of the main actors, namely the Planning and Finance currently is in the design and development phase. Its ministries, the Office of the President, sector ministries, objectives are to improve public management by provid- and Congress. It is also critical to create the right ing more information on the results achieved through incentives to encourage these stakeholders to assume a public expenditures. The performance indicators that are greater role in M&E. being developed at different levels of the national · Strengthening M&E systems is not only, nor even princi- government (sectors, agencies, organizational units, etc.) pally, a supply-side issue requiring a "technical fix". For 3 an M&E system to be considered successful and sustain- are envisioned only as facilitators, so that country able, the information and findings of M&E have to be ownership is preserved and assured. utilized intensively by all stakeholders, including sector ministries and, depending on how the system has been devised, civil society. · Conservative ministries and staff may resist efforts to "Country efforts to strengthen their M&E implement M&E systems and to use M&E as a manage- systems can be a catalyst for broader public ment and budget tool. It is essential to have a high-level of sector reform. The Bank acknowledges a commitment and on-going support from powerful champi- clear demand for a region-wide M&E net- ons at the ministerial and senior official levels. work." · There is an implicit debate on how to prioritize evaluations ­ to focus on "problem" programs, pilot projects, high- -Ernesto May, Sector Director, LAC Poverty expenditure or high-visibility programs, or on systematic Reduction & Economic Management, The research to respond to questions of program effective- World Bank ness. Informal M&E Network for the LAC Region The conference proceedings will be published and disseminated in both Spanish and English, and will be It was evident from the June conference that a growing available from the network's planned website: number of LAC countries are initiating efforts to www.worldbank.org/evaluation/LACMandE. strengthen and systematize their M&E functions. Senior officials from the eleven countries represented at the Notes conference found highly valuable the sharing of their country experiences in institutionalizing M&E. They were impressed by the substantive progress achieved in several 1Chile: Study of Evaluation Program Impact LAC countries in this field. Evaluation and Evaluations of Government Programs, The World Bank, 2005 2Guzmán, Marcela, Systems of Management Control "Truly the stars and the planets have aligned and Results-based Budgeting. The Chilean in terms of the interest in this subject [institu- Experience, Ministry of Finance, National Budget tionalizing M&E systems], both within coun- Office, Management Controls Division, (available at tries and in the international community. www.dipres.cl/fr_control.html) This is an exceptional window of opportu- 3The World Bank Report (an evaluation of the Chilean nity." evaluation system) has been commissioned as part of advisory services provided by the World Bank in Chile - Manuel Fernando Castro, Head of Evaluation, (Task Manager Fernando Rojas, LCSPS). For full National Planning Department of Colombia reference, see footnote 1. 4Oportunidades, previously known as Progresa, is a conditional cash transfer program that provides monthly cash assistance to poor families if they give their children The proposal was, therefore, made at the conference to access to specific educational and health services. create an informal network of key stakeholders, to focus on the institutionalization of M&E and to facilitate South- South learning throughout the region. It was proposed that the network, which will be the first of its kind in the About the Authors developing world, will be open and flexible, attracting This note was prepared by Ruxandra Burdescu, LAC decision-makers, practitioners, experts in and outside the Public Sector Consultant; Azul del Villar, LAC Public government; high-level officials from sector ministries, Sector Consultant; Keith Mackay, Senior Evaluation finance ministries, and planning departments; parliamen- Officer in the Operations Evaluation Department; tarians and their advisors; academics; consultants and Fernando Rojas, LAC Lead Public Sector Manage- allow them to share their own country experiences with ment Specialist; and Jaime Saavedra, Sector Manager each other, and to provide them with additional regional of the LAC Poverty and Gender Group. and some global experience. The World Bank and IDB 4