56364 The World Bank August PREMnotes 2010 NuMBER 1 Special Series on Conceptual Framework for Monitoring and Evaluation Keith Mackay Series Introduction This is the first note in a monthly series on government monitoring and evaluation (M&E) systems led by the PREM Poverty Reduction and Equity Group under the guidance of Jaime Saavedra, Gladys Lopez- Acevedo, and Keith Mackay, with contributions from several World Bank colleagues. The main purpose of this series is to synthesize existing knowledge about M&E systems and to document new knowledge on M&E systems that may not yet be well understood. The series targets World Bank other donor staff who are working to support client governments in strengthening their M&E systems, as well as government officials interested in learning about the uses and benefits of M&E and in adopting a more systematic approach toward M&E in their governments. A growing number of governments in devel- increasing welfare, reducing poverty, and im- oping countries around the world are work- proving opportunities for all. An additional ing to improve their performance by creating impetus to focus on performance is the systems to measure and help them under- strong expectations of international donors. stand the performance of their services and There is an increasing body of literature programs. This trend is influenced by Organ- on the experience of countries in building isation for Economic Co-operation and De- and strengthening their M&E systems, and velopment (OECD) countries, most of which on the M&E tools and techniques that they place a high priority on the four main uses of use. This literature comprises conference M&E findings: proceedings, research papers on country 1. policy development case studies, academic reviews, and numer- 2. evidence-based policy making and budg- ous Web sites of donors, governments, and eting evaluation associations. Accessing and digest- 3. management performance ing the information in its various formats can 4. accountability. be difficult, especially for senior officials who The priority for measuring and better are new to M&E. managing government performance in mid- dle- and low-income countries is intensified Why M&E Systems Improve by continuing fiscal and macroeconomic Government Performance pressures affecting all countries, and by ever- This note1 outlines the main ways in which rising expectations from ordinary citizens. It M&E findings can be used throughout the is also influenced by the need for citizens, policy cycle to improve the performance of governments, and the international commu- government decision making and of govern- nity to make state actions more effective in ment services and programs, including the FROMtHEPOVERtYREDuCtIONANDECONOMICMANAgEMENtNEtWORK use of M&E for evidence-based policy mak- levels. It can identify what works, what does- ing, budgeting, management, and accounta- not, and the reasons why. M&E also provides bility. There are many different types of M&E information about the performance of a gov- tools and approaches, each with advantages ernment, of individual ministries and agen- and limitations. This note presents four ex- cies, and of managers and their staff. High- amples of successful government systems for lighting examples of good practice and poor M&E--in both developed and developing practice can help improve performance. countries--and discusses some of their hard- Three defining characteristics of success- earned lessons for building M&E systems. ful M&E systems are These lessons are evidence of what works 1. intensive utilization of the M&E infor- and what doesn't in the development and mation in one or more stages of the pol- sustainment of successful M&E systems. icy cycle; Government systems for M&E focus on 2. information that meets standards for data measuring the results produced by govern- quality and evaluation reliability; and ment--its outputs, outcomes, and impacts. 3. sustainability, by which the system will The M&E system may exist at the level of an survive a change in administration, gov- individual agency, entire sector, or the gov- ernment ministers, or top officials. ernment as a whole. M&E can provide Perhaps the best way to understand the unique information about the performance potential contribution of M&E to sound gov- of government policies, programs, and proj- ernment is to view it at different parts of the ects--at the national, sector, and subnational policy cycle (figure 1). The early stages of the Figure 1. The Policy Cycle: Linking Policy, Planning, Budgeting, Management, and M&E Review Policy Review the past planning and budget period Develop Policy Evaluate and Audit and Begin Planning The efficiency and Set objectives, policies, effectiveness of activities strategies, and priorities Monitor Activities Prepare Budget and Account for Allocate resources Expenditures Implement and Manage Planned Activities Source: Adapted from World Bank (1998). 2 PREMNOtE August 2010 policy process--the analysis and develop- particularly important. Comparisons made ment of government policy and the planning over time can help identify good, bad, and of priorities and strategies, including how promising practices. Evaluations or reviews consistent are they with the objective of can identify the reasons for this good or bad poverty reduction--all benefit from evidence performance. This is the learning function of of what has or has not worked in the past; in M&E, often referred to as results-based man- other words, evidence-based policy making. It agement. is an important discipline for governments to The final stages of the policy cycle in- consider carefully what they are trying to clude accountability relationships. M&E re- achieve from their policies, and to plan for veals the extent to which the government has them accordingly. Thus it helps to clarify, for achieved its objectives and thus provides the each possible program, what success would evidence needed to ensure strong accounta- look like and how the government will know bility, such as of government to the congress if it has (or has not) been achieved. It also or parliament, to civil society, and to donors. helps to clarify what are the potential poverty M&E also supports accountability relation- and distributional effects of policies and pro- ships within government, such as between grams. Setting performance targets, and sector ministries and central ministries, measuring the extent of progress toward among agencies and their sector ministry, achieving them, is thus an important part of and among ministers, managers, and staff. government planning and policy review. Strong accountability can provide the incen- Information on the performance of ex- tives necessary to improve performance. isting government programs, and on the ex- M&E can also play a role in anticorrup- pected performance of new programs, is im- tion efforts. It can help identify "leakages" in portant for the next stage of the policy cycle: government funding, as well as some of the budget decision making--the allocation of possible manifestations of corruption--such resources in the budget. M&E information, as when government spending is not reflect- and especially evaluation findings that ex- ed in the physical quality of infrastructure or plain past performance, helps to guide gov- in the volume and quality of government ernment decisions so that the most cost-ef- services provided. fective collection of policies and programs can be adopted in the annual budget. What Is M&E? At the next stage in the policy cycle--the To some senior officials and donor staff, M&E implementation and management of activities can appear to be a highly technical topic with funded by the budget--M&E helps managers techniques that at first glance can be difficult to monitor their activities, including govern- for nonspecialists to understand. To make ment service delivery and staff management, matters worse, M&E contains a lot of buzz so that they learn quickly what is working and words and terminology that are confusing to what is not, for example, in terms of expected those not in the field. Sometimes the phrase outputs, expected outcomes, or even higher "M&E" is identified with specific tools, and level objectives such as increasing welfare. Per- means different things to different people. formance indicators can be used to make cost Technical and semantic debates are unimpor- and performance comparisons--performance tant to the readers of this series; what is im- benchmarking--among different administra- portant is the appropriate use of the key M&E tive units, regions, and districts. Ongoing mon- tools and techniques that can measure the itoring of these activities--including spending, performance of government programs in dif- processes, outputs, outcomes, and impacts--is ferent ways. Some commonly used tools in- August 2010 PREMNOtE 3 clude performance indicators, rapid evalua- There is no simple answer to the com- tions, impact evaluations, and comprehensive mon question: what does a good M&E system spending reviews. A forthcoming note in this look like? Rather, it all depends on country- series will discuss M&E tools. specific factors, such as government demand for M&E information; the uses to which the What Does a Government System information will be put; the availability and for M&E Look Like? quality of existing government data; the abil- Most governments have data systems for ities of officials and consultants to conduct measuring their spending, processes, and evaluations and to analyze M&E information; outputs. But this is not the same as a system to and the amount the government is prepared monitor and evaluate the performance of all to spend on M&E. of its programs. A much smaller number of governments possess such systems. These sys- The Three Defining tems involve the regular, systematic collection Characteristics of Successful and use of M&E information; an M&E system M&E Systems may exist at the level of an individual agency, A "successful" M&E system has three de- or an entire sector, or for the government as a fining characteristics. The first is intensive whole. The focus of these systems is on meas- utilization of the M&E information provided uring the results produced by government-- by the system in one or more of the stages of its outputs, outcomes, and impacts. the policy cycle discussed above. It may seem Fortunately, there are a number of con- trite to argue that M&E information should vincing, well-documented examples of gov- only be collected if it is going to be used, but ernments that have devoted the effort neces- most evaluators in governments (and in sary to build high-performing M&E systems. donor agencies) have a surprisingly poor un- Examples of four countries with well-docu- derstanding of the extent to which the M&E mented and analyzed M&E systems are Aus- information they produce is actually used by tralia, Chile, Colombia, and the United States others. If M&E information is not being used (see box 1). Each of the four governments uti- then it is important to discover the reasons lizes its M&E information intensively. Other why. Is it because the M&E information is re- common features include a powerful central garded as being of poor quality, or not ministry with leading role in the M&E system timely, or because evaluations have not ad- (such as the finance or planning ministry), dressed the most relevant questions concern- and--as a consequence--an emphasis on us- ing program performance? Or is it because ing M&E information to support the budget the intended users within the government-- process. What is less well known and docu- such as the finance or planning ministries-- mented, however, is how effective these sys- have neither the skills nor interest in using tems have been in supporting the use of M&E this information in their work? information for ongoing program manage- Reliable, quality information is another ment. Of particular note are the differences feature of successful M&E systems. There between these four M&E examples. Different are various standards of what constitutes countries emphasize different M&E tools and quality monitoring data and evaluations, and techniques. Some place a heavy reliance on these standards can be used to assess the reli- monitoring data, others on evaluations and ability of the information that any M&E sys- reviews of various kinds. Some countries have tem produces. Most government evaluation extensive (and expensive) M&E systems; oth- offices have some sort of quality control ers have more streamlined, low-cost systems. mechanism in place. Most, however, do not 4 PREMNOtE August 2010 Box 1. Four Examples of Successful Government M&E Systems Australia: The government evaluation system was managed by the Department of Finance (DoF), and it re- quired ministries to evaluate every program every three to five years. The evaluations were conducted by the line ministries themselves, but they were overviewed by the DoF and other central departments. By 1994, al- most 80 percent of new spending proposals in the budget process relied on evaluation findings, usually to a significant degree. About two-thirds of savings options also relied on evaluation findings. DoF officials, who at- tended the Cabinet meetings that considered these budget proposals, judged that this information was highly influential on the Cabinet's budget decision making. The Australian National Audit Office found that line depart- ments also used this information intensively, particularly to help improve their operational efficiency. Chile: The Ministry of Finance (MoF) commissions the evaluations externally to academics and consulting firms, and it uses standardized terms of reference and methodologies for each type of evaluation it uses. MoF officials use the M&E findings intensively in their budget analysis of the performance of each ministry and agency. The ministry also uses the information to set performance targets for each agency and to impose man- agement improvements on both ministries and agencies. The MoF carefully oversees the extent to which each ministry implements these management improvements. Colombia: The government's M&E system, SINERGIA, is managed by the Department of National Planning. The system includes information for 500 performance indicators, as well as a number of rapid and impact eval- uations. The president has used the information provided by SINERGIA intensively in his monthly management control meetings with each minister and in his weekly town hall meetings in municipalities around the country. United States: In 2002, the government created the Program Assessment Rating Tool (PART), building on earlier efforts to measure government performance. The performance of all 1,000 government programs have been rated using the PART methodology, and PART ratings are required to be used by departments in their annual budget funding requests to OMB. The requests must highlight the PART ratings, the recommendations for improvements in program performance, and performance targets. OMB, in turn, also uses the PART ratings when it prepares the administration's funding requests to the Congress, and to impose performance improve- ment requirements on departments. Source: Mackay 2007 appear to conduct or commission formal re- ed by donors rather than by the government views of the quality of their work. Three of itself, then sustainability is less likely. the four countries highlighted in box 1 have conducted such reviews: Australia, Chile, and Building a Government M&E Colombia. System--What to Do and The third characteristic of a successful What Not to Do M&E system is sustainability. This relates to Many developed and developing countries the likelihood that the M&E system will have accumulated substantial experience in survive a change in administration or in gov- building M&E systems. As with any form of ernment ministers or top officials. When the capacity building, there are a number of utilization of M&E information is firmly em- hard-earned lessons about what works best bedded--that is, mainstreamed--in core gov- and what does not (discussed more fully in ernment processes, such as the budget cycle, Mackay [2007]). Eight key lessons are dis- it can be said to be institutionalized and thus cussed below. is likely to be sustained over time. Conversely, Lesson 1: First and foremost is the need when M&E has only a handful of key sup- for substantive government demand for porters or is little used, or if it is largely fund- M&E information. Such demand is necessary August 2010 PREMNOtE 5 if a serious effort to build an M&E system is carrot is the provision of greater autonomy to be started and sustained. A significant ef- to managers who can demonstrate (through fort is required to build an M&E system, in- reliable M&E information) that their pro- cluding the creation or upgrading of data grams are performing well. An example of a systems such as decisions about types of data stick is to set challenging (but realistic) per- to be collected, data collection methods, formance targets that each ministry and pro- storage, quality control, and transmission. gram manager is required to meet. An exam- Equally important components of the M&E ple of a sermon is a high-level statement of system include the training of statistical ana- support for M&E, such as from a president lysts; choice of evaluation tools and tech- or influential minister. Many of these incen- niques, and their adaptation to local circum- tives have been applied successfully in build- stances and priorities; training of evaluators ing M&E systems in developed and develop- and development of national evaluation con- ing countries. sultants; creation of M&E offices inside a Lesson 3: It helps to start with a lead ministry and probably in some or all sec- diagnosis of what M&E functions already ex- tor ministries; training of the users of M&E ist in the country--in the government, aca- information--mid-level analysts, senior offi- demia, and the consulting community. A di- cials in central and sector ministries, and pos- agnosis should identify the strengths and sibly their ministers; and the creation of a bu- weaknesses of what exists on both the de- reaucratic infrastructure to decide which mand and supply sides. A diagnosis is really a government programs should be evaluated type of evaluation, and the very process of and what issues should be addressed in each conducting it provides an opportunity for evaluation. Frankly, this effort is not worth- key stakeholders within the government to while unless the resulting M&E information become more familiar with M&E and its po- is likely to be used intensively. tential benefits to the government. A diagno- Lesson 2: Incentives are a key part of the sis naturally leads to an action plan to demand side. Strong incentives are needed strengthen M&E. This can facilitate a coali- for M&E to be conducted, and for the infor- tion of support from interested sector min- mation to be used. M&E experts often make istries and the donor community. a basic mistake by asserting that M&E infor- Lesson 4: Another dimension of the de- mation is intrinsically "a good thing" and mand side is the need for a powerful champi- that if the information is made available, on, an influential minister or senior official then it will automatically be used. This tech- who is able to lead the push to institutionalize nocratic view that M&E has inherent merit is M&E, to persuade colleagues about its impor- naïve; M&E information has value only if it is tance, and to allocate significant resources to reliable and if it is used intensively. creating a whole-of-government M&E system. Utilization does not usually--and does Government champions have played pivotal not regularly--happen by chance. There need roles in some of the most successful M&E to be incentives for M&E information to be systems. used by program managers in their day-to- What has been a lot less successful is re- day work, by budget and planning officials liance on a law, decree, or cabinet decision to responsible for advising on policy options, or institutionalize an M&E system. Such an ap- by a congress or parliament responsible for proach can help to legitimize an M&E sys- accountability oversight. tem, particularly in those countries where There are three types of incentives: car- the presence of a legal instrument is viewed rots, sticks, and sermons.2 An example of a as necessary for any government reform to 6 PREMNOtE August 2010 be perceived as worthwhile and to be taken and agencies. In Mexico, for example, the so- seriously. But a law or decree on its own does cial development agency (SEDESOL) had not ensure that the considerable efforts re- eight different, uncoordinated management quired to build an M&E system will be under- information systems. In Uganda in recent taken and maintained. years, there were as many as 16 separate sec- Lesson 5: Another common feature of tor and subsector systems that the govern- successful M&E systems is the stewardship by ment had to try to coordinate. a capable ministry that can design, develop, That said, there is real value in building and manage the system. Thus it helps to have reliable ministry data systems: these provide the institutional lead of the M&E system the raw data on which a whole-of-government close to the center of a government, such as M&E system depends. An audit of data sys- in the president's office or in the finance or tems and a diagnosis of data capacities can planning ministries. be helpful in this situation because they One role of this institutional leader is to could provide a basis for rationalizing exist- continually review the extent of progress in ing data collections and improving their developing the M&E system and make any quality. necessary adjustments to its action plan. Dif- Lesson 7: This is, unsurprisingly, that ficulties and roadblocks are inevitable, so it is building an M&E system usually includes important to identify what is working, what is training for a range of M&E tools, methods, not, and why. Regular reviews of progress approaches, and concepts. It is necessary to provide the opportunity to analyze both the have well-trained officials or consultants who demand and supply sides of the M&E system. are highly skilled in M&E. Training should In fact, most countries with well-performing provide more than competencies in M&E, M&E systems have not actually developed however. Senior officials need to understand them in a linear manner--that is, starting the strengths and limitations--the relative with a clear understanding of what the cost-effectiveness--of various types of M&E system would look like once fully mature and tools and techniques. Introductory training then progressively achieving this vision. can also raise awareness of and demand for Rather, countries' M&E systems are more M&E information. Training should also ex- commonly developed incrementally and tend to the use of M&E findings. Budget an- even in a piecemeal and opportunistic man- alysts, poverty analysts, and program man- ner, with some false starts and blind alleys agers need to be able to interpret monitoring along the way.3 data to understand trends, data definitions, Lesson 6: A common mistake once M&E breaks in data time series, and so forth. They has been embraced enthusiastically is to over- also need to be discriminating consumers; engineer the M&E system. This is often evi- they must be able to tell when an evaluation dent in the large number of performance in- is reliable or when its methodology or find- dicators that are collected. Over-engineering ings are questionable. can also result in the proliferation of ministry Lesson 8: A final lesson is that it requires data systems. These are often uncoordinated a long-term effort, with patience and deter- even within each ministry. The problem is mination, to build an effective M&E system. multiplied if there are several whole-of- It takes time to create or strengthen data sys- government data systems, which may be tems; to recruit and train qualified staff; to managed by different central ministries and plan, conduct, and manage evaluations; and may well require related (but different) in- to train staff to use M&E in their day-to-day formation to be provided by sector ministries work, whether that involves program opera- August 2010 PREMNOtE 7 tions or policy analysis and advice. Australia the Bank in 1997, he was the manager of Australia's and Chile were able to create well-function- whole-of-government evaluation system. He has written ing M&E systems--in terms of the quality, over 60 publications on M&E systems. number, and utilization of evaluations--with- in four or five years; but in Colombia's case, Endnotes it has taken more than a decade. 1. This note has benefited from comments by Gladys Lopez-Acevedo, Nidhi Khattri, Jaime Saavedra, and Helena Hwang. The views ex- Conclusions pressed here are those of the author. A growing number of developing countries 2. A long list of M&E incentives is provided by are successfully building government M&E Mackay (2007, chapter 11). systems. They look to the examples of devel- 3. Reasons for this are discussed more fully by oped countries--especially members of the Mackay (2007). OECD--but increasingly they are also looking to their peers: countries such as Chile, Colom- References bia, and Mexico. These countries have cre- IEG (Independent Evaluation Group). 2004a. "In- ated well-performing M&E systems where the fluential Evaluations: Evaluations That Im- M&E findings that are produced are used in- proved Performance and Impacts of Develop- tensively. These countries have demonstrated ment Programs." World Bank, Washington, not only that it is feasible to build a govern- DC. ment M&E system, but that the systems are ------. 2004b. "Monitoring and Evaluation: also valued highly by the governments. Some Tools, Methods and Approaches." 2nd edition. World Bank, Washington, DC. Hopefully, this short note and the future Lopez-Acevedo, Gladys, Katia Rivera, Lycia Lima, notes in this series will engage the interest of and Helena Hwang, eds. 2010. "Challenges in senior officials in developing countries and Monitoring and Evaluation: An Opportunity prompt them to fully investigate whether to Institutionalize M&E Systems." World Bank their government should devote the time and Inter-American Development Bank, and effort to building such a system. The Washington, DC. donor community--including the World Mackay, Keith. 2007. How to Build M&E Systems to Bank--stands ready to support them in this Support Better Government. Washington, DC: work. World Bank. World Bank. 1998. Public Expenditure Management About the Author Handbook. Washington, DC: World Bank. Keith Mackay retired from the World Bank in 2008 World Bank Web Sites and now works as a consultant. He was a lead evalua- www.worldbank.org/poverty tion officer in the Bank's Independent Evaluation www.worldbank.org/lacmonitoringandevaluation Group, providing technical advice and other support to www.worldbank.org/ieg/ecd countries working to strengthen their M&E systems as an integral part of sound governance. Before joining This note series is intended to summarize good practices and key policy findings on PREM- related topics. The views expressed in the notes are those of the authors and do not necessarily reflect those of the World Bank. PREMnotes are widely distributed to Bank staff and are also available on the PREM Web site (http://www.worldbank.org/prem). If you are interested in writing a PREMnote, email your idea to Madjiguene Seck at mseck@worldbank.org. For addi- tional copies of this PREMnote please contact the PREM Advisory Service at x87736. PREMnotes are laid out by UpperCase Publication Services, Ltd. Prepared for World Bank staff