64382 Harry Hatry This paper1 identi�es key steps in designing and implementing a monitoring and evaluation (M&E) system for ministries and individual government agencies that provide services. These suggestions are intended to apply whether the ministry or agency is in health, education, social welfare, environmental protection, transportation, economic development, public safety, or any other sector. The system might have been ordered or requested by the president or prime minister’s office, by a minister, or by any agency head who wants to implement an M&E process. M&E development should focus on providing a process that will yield regular outcome data (in addition to data on the organization’s outputs) that can be used by the designing agency and upper-level officials for accountability and, particularly, for managing these organizations, thereby helping officials improve their accountability and services to their citizens. 1. Establish a committee of key agency per- There are a number of basic conditions that are sonnel to guide and oversee the implementation vital to successful M&E implementation. First, effort. The committee should include individuals there is a need for strong, top-level supporting from within the organization as well as representa- leaders who are willing to provide the time and tives from program offices; the budget office; the resources to initiate the M&E process. Second, the planning office; the information technology office; presence of at least some staff and enough data col- the human resources office; and an analytical of- lection capacity to undertake basic collection and �ce, such as a statistical office, if one exists, whose analysis are critical. Third, government officials work might be expected to relate to M&E. It is also must be willing to use the M&E data to help make likely to be helpful to include in the committee an resource allocation decisions, and not merely “outside� person who can bring a broader perspec- use the data to show that the agency is collecting tive, such as someone from a higher-level budget performance data. In addition, there should be or planning office. enough flexibility for agency managers to make 2. Select a small cadre of support person- changes where the data indicate that changes are nel who will be responsible for supporting the needed. If these conditions are absent, the M&E implementation of the various administrative and process may be a waste of time. technical elements of the M&E system (under the guidance of the committee). 3. Consider meeting in the early stages with There are a number of basic steps for designing members of the legislature to find out what and implementing the process, whether at the information would be useful to them for annual ministry or agency level. budget deliberations or other policy or oversight purposes. Similar input should be sought from needed. For example, a ministry should review the national budget office. the indicators selected by each of its agencies. An 4. Develop a plan and schedule for implemen- agency should review the indicators proposed by tation keeping in mind it is likely to take three each of its programs. to �ve years for full implementation. However, 8. Inventory existing databases within the some initial useful performance information can agency to identify the output and outcome in- probably be expected to become available by the formation already available (or at least readily end of the �rst year. available) that can be used to monitor progress As part of this step, it is a critical to identify toward the agency’s mission. The sources might be who will be needed to undertake each step of inside or outside the agency, such as data already implementation. The people selected should be regularly collected by a central planning office. available and committed to providing the time Many, if not most, countries and their agencies necessary to perform their assignments. already regularly collect outcome data such as 5. Provide training and technical assistance incidence and prevalence of health problems to managers and key professional staff in the (for example, malaria, tuberculosis, and HIV); basics of M&E. This applies to initial training road accident fatalities; drinking water and air and for training new people as turnover occurs. quality; unemployment rates; school absenteeism The training for managers should provide a thor- and completion rates; and economic development ough understanding of what M&E entails and indicators and so forth. recommendations for using the information. The 9. For each candidate performance indicator, training for other staff should focus more on the identify the source of the data, particular data details on how M&E should be conducted so as collection procedures, and how often the data will to make the make the information obtained both be collected and reported. Some outcomes will valid and useful. Multilateral and bilateral fund- require new data collection. Note that an indica- ing organizations are potential sources of support tor should not be considered �nal until a feasible for these activities. data collection procedure has been identi�ed. An 6. Ask each program manager within the agency and its programs should not limit itself to agency to identify a results-focused mission state- considering only outcome indicators for which it ment for their programs. These statements will currently has data. Depending on the availability necessarily be general, but they should set the tone of resources, however, the agency may need to and direction for the M&E effort in each program. defer beginning new data collection procedures 7. Ask each program within the agency to until later years. select a set of outcomes and outcome indicators In addition to data obtainable through agen- that will enable the program to monitor progress cies’ records, two other major data collection op- in achieving the results identi�ed in its mission tions to consider are customer surveys and trained statement. Programs could establish their own observer ratings. working groups of key personnel to help select Surveys of citizens. Surveys are increasingly be- appropriate outcomes and outcome indicators. ing used around the world to obtain feedback on The M&E working group might also seek input outcomes, including the quality of government from representatives of the customers served by services. The feedback can be from samples of the program. For example, the program might households or of customers that have been as- sponsor meetings (sometimes called focus groups) sisted by the agency (such as patients, students, or with small numbers of its clients to help identify businesses). These surveys can provide feedback outcomes of importance to them. A public welfare on a variety of program outcomes, including the agency, for example, might hold such sessions surveyed persons housing, employment, or health with its clients; a health agency might hold such condition status; ratings of various program qual- sessions with patients in health clinics; and an ity characteristics, such as the program’s timeli- economic development agency might hold such ness, accessibility, fairness, and ease of obtaining sessions with representatives from businesses. information on the program’s services; frequency The set of outcome indicators should be of use and reasons for nonuse of the service; and reviewed at a higher level to ensure that the set extent of having to make inappropriate payments contains the important performance information to obtain service. Citizen surveys can also obtain information on needed program improvements and training or the demographic characteristics of respondents, technical assistance needs, and to draw attention which helps identify the distributional effects of to successful programs. Using M&E data at all the program. levels of the agency, such as planning, budgeting, Trained observer rating procedures. This option and program review, highlights the importance of uses personnel to systematically rate physically the M&E system. observable conditions such as the condition of 12. Review the monitoring data and data roads, hospitals, health clinics, school buildings, collection procedures periodically. This review or other public facilities; proximity of households is an appropriate time to introduce improved to sources of drinking water (such as standpipes); measurements to ensure the continued reliability and the ability of persons with disabilities to and credibility of the performance measurement undertake basic activities of daily living. Each of work. Performance measurement should always these conditions is rated by an observer trained in be considered to be a work in process. the use of a well-de�ned rating scale. The raters 13. Use the M&E information to lay the visit all, or samples of, the units to be observed, groundwork for positive incentives for improv- such as the roads, hospitals, or other facilities. The ing the effectiveness of public services. From the outcome indicator would be the “number and beginning, emphasize that achieving good results percent of those items rated as being in satisfactory will be rewarded. Emphasize that a major purpose condition.� Ratings obtained at regular intervals of M&E information is to contribute to the learn- enable the program to monitor the changes in each condition. ing process to help officials and their staff identify The procedure has a potentially very useful what works and what does not. side bene�t to program operations. The ratings 14. Emphasize the message that outcome of individual facilities or other units that are information is highly useful for telling what the inspected can be used to identify where work is running outcome score is, but does not provide needed and the extent of that need (for example, the background information that tells why. The is the condition hazardous) so those units with M&E system should not be used as a blame game. poor ratings (that is, are in poor condition) can M&E information is often perceived by managers be prioritized for correction as soon as possible. as threatening because it has been used to blame 10. Provide for basic analysis of the perfor- managers for poor outcomes without considering mance data. At the ministry or agency level, why the outcomes occurred. Outcomes are often top-level officials should assign an office to be affected by factors over which managers have responsible for examining the data collected, limited or no control. identifying unexpected �ndings, and reporting these �ndings to the appropriate officials in time to allow the agency to act on them. The basic analysis for performance monitor- Following are six key ways to maximize the useful- ing is typically comparisons. Such comparisons ness of an M&E process within a service agency: include comparing the current reporting period 1. Link key characteristics of the “customers� to previous reporting periods; comparing cur- the agency/program is serving to the outcomes for rent data to targets; comparing outcomes among different customer groups; and comparing the each customer. Then, report the outcome data for �ndings to those in similar programs in other each such category of customers as well as for all countries. While more sophisticated statistical customers. This information can be highly useful analysis would be desirable, this �rst-level, basic both for interpreting the data and for identifying analysis should be highly informative and make what actions are needed. the monitoring considerably more useful. For example, program managers in the educa- 11. Emphasize the importance of outcome tion, employment, public welfare, housing, and information, particularly by using the M&E in- health sectors will �nd it very useful to be able to formation in decision making. Top-level officials identify program outcomes by demographic char- in the agency also need to use the monitoring acteristics, such as outcomes by age, gender, in- information collected on outcomes to identify come, household size, race/ethnicity, and so forth. This may suggest the need for program changes for than X percent of the expected target value, such groups that were not sufficiently helped. as more than 10 percent. For programs for which people are not direct Data on results do not tell what the cause of service recipients, such as road maintenance and those results is, or what to do about them. Agen- environmental quality programs, breakouts by cies and their programs should be encouraged key characteristics are still very important. For to seek explanations for unexpected results and example, data calculated on number of traffic develop plans to correct any problems that are accidents will be considerably more useful if the identi�ed. A side bene�t of such a provision is outcome data are broken out by characteristics that managers may welcome the opportunity to such as cause of the accident, average daily traffic, formally provide their explanations, fearing that and location. otherwise they will be blamed unfairly for poor A key problem for many countries, particu- results. This may alleviate the “blame� problem larly poor countries, is that they may not have MIS noted earlier. systems capable of providing such information. 4. Provide for regular, timely performance Nevertheless, the programs in such countries reporting for review by ministry, agency, and should do whatever they can to try to obtain and program officials. For budgeting purposes, only calculate such breakout data, even if the calcula- annual performance reports may be needed. tions only can be done manually, and even if they However, for managerial purposes, this is too in- can only focus on outcome breakouts for one or frequent. Generally, reporting should be done at two categories. least quarterly. For some indicators, such as those 2. Set targets for the coming year for each relating to safety and health, reports are likely to output and outcome indicator. A good time to be needed more frequently. select targets is during the annual budget process. 5. Make the report format as understandable, Choosing these targets is more of an art than a readable, attractive, and informative as possible. science. Setting targets typically starts with the Too often performance reports are unclear or data on the previous years’ values for each indi- uninformative or �lled with unfamiliar termi- cator. These values then can be adjusted for any nology. With today’s electronic capabilities, even signi�cant changes expected, such as in funding poor countries have access to a variety of report- or personnel. Other considerations are external generating software. factors, such as the economy (such as changing 6. Ministry officials, agency officials, and export conditions or new businesses that can be program managers should be encouraged to hold expected to increase business revenues or reduce regular “How Are We Doing?� review sessions unemployment); technology (that is expected soon after the latest performance report has been to lead to improvements in service quality); and issued. The ministry or agency head meets sepa- political changes expected to alter ministry or rately with each major agency program or division agency priorities. to discuss the performance data, identify where Normally the program should propose the the outcomes are going well and where not well, targets for its own performance indicators. These and suggest ways to correct poor past results. In targets, however, should be reviewed by upper- subsequent meetings, previous decisions should level officials to ensure that the targets are neither be followed up on to determine the extent of prog- too easy to meet nor overly high. The danger with ress that has been made in correcting previously targets is that they can become primarily political identi�ed problems. in nature, making the targets either much too easy to achieve or unrealistic. Targets are more mean- ingful when based on previous performance and reasonable expectations of the future. 3. Ask agencies and their programs to provide The monitoring process described above provides explanations for unexpected results and indicate regular measurement of the performance of pro- plans for correcting any problems noted in the grams. Such information does not tell why the explanations. The ministry or agency might set a program has been successful or unsuccessful. The threshold such as requiring explanations for any explanatory information suggested above provides performance indicator value that differs by more �rst order information on the “whys� and what might be done to improve performance. However, to complete the evaluation and therefore the for programs that involve large expenditures and/ timeliness of the �ndings; and likelihood that or are of major importance, more in-depth pro- the evaluation will provide useful information gram evaluation is desirable. In general, program for later decisions. For some programs it may not evaluations tend to be expensive and may require be feasible to obtain the needed information. An long periods of time, perhaps years before the initial step is to assess whether these programs are �ndings are available. good candidates for evaluation. Each year, it is important for the ministry It is also critical to select who (what type of and each agency to establish a schedule of which organization) will conduct each evaluation. The programs will receive an in-depth evaluation. The organization conducting the evaluation should choice should depend on criteria such as major be reasonably independent, if for no other reason uncertainties and lack of information regarding than to achieve external credibility for the �nd- program success; cost of evaluation; time required ings. Few agencies or ministries governments are likely to have the in-house capability for such could be given to agencies or programs that have program evaluations. Almost certainly the evalua- met (or exceeded) their targets. tions will need to be conducted by outside organi- M&E information provides the basis for de- zations such as universities or specialized private veloping multiyear strategic plans for ministries nonpro�t or for-pro�t organizations. Because of and agencies. The latest values for key outcome their costs, the ministry or agency might seek indicators can be used to establish the baselines funding from one of the international multilateral and subsequent out-year targets. In later years, or bilateral funding organizations, particularly the annual data from the M&E process on these for evaluating programs for which the funding key outcome indicators can be compared against organization has provided support. the targets in the multiyear strategic plan. Such An important additional step is to ensure information indicates whether actions are needed that the completed evaluations are professionally to stay on the plan, whether plan revisions are reviewed to assess their quality. Most program needed, or whether the plan is no longer feasible. evaluation reports should be made available to The information produced also helps to the public. This step itself can help encourage communicate with the legislature and citizens. the program evaluators to ensure their work is of Performance reports that provide data on the reasonable quality. ministry’s or agency’s major outcome indicators M&E information has many uses, including identifying shortfalls in outcomes, thereby en- abling service improvement. M&E data can also help to formulate ministry, agency, and program budgets and help to justify budget requests. Pro- gram outcome information, whether obtained through regular performance monitoring or from program evaluations, should be linked to program cost information as part of the budget process. The outcomes expected to be achieved by the budget requested should be a major factor in budget decisions. The information produced by M&E systems is also useful for helping to allocate and prioritize re- sources throughout the year, such as adjusting the assignment of work or staff to different locations or customer groups based on performance data. For example, outcome indicator data can identify: areas most in need of road repairs; emerging health issues; types of housing shortages; traffic accident locations; and locations with the most water or air quality problems. M&E systems can also develop incentives for agencies and programs based at least in part on success in achieving outcomes. Monetary incentives can be expensive and are not likely to be appropriate for many countries—at least not until considerable experience in M&E has been gained. However, nonmonetary incentives can be considered once the M&E process is in place. For example, as noted earlier, recognition awards are likely to be of considerable interest to legisla- university, or the national statistical office. Added tors and citizens. Such transparency can be helpful costs could also include costs for a new computer in obtaining citizen support for the ministry’s and system and technology to process the substantial agency’s work. amounts of data, as well as in-depth program Ministries and their agencies may, under- evaluation costs. In-depth program evaluation is standably, fear that showing bad news (such as usually conducted by an outside organization such worsening outcomes) will only bring them grief as a university or specialized private �rm. from the legislature and public. However, showing The position of many governments in devel- only outcomes that look good is likely to lead to oped countries that are introducing M&E systems a lack of credibility. Problems can be alleviated if has been that monitoring is a basic management the ministry or agency, through their programs, function. The cost of ongoing implementation provides explanations for poor outcomes and in- should, therefore, be primarily covered by the dicates what it plans to do to correct the problem. ministry, agency, and the program’s own budget. Initial start-up costs are likely to be more of a problem for many ministries and agencies, how- ever, and for these costs funding could be sought Depending on the level of decentralization in the from the central government or from donor country, a considerable amount of the data may organizations. come from local or regional/provincial govern- ment bodies. For example, to the extent that edu- cation and health services are decentralized, these lower levels of government should be monitoring Monitoring and evaluating the results of public the outcomes of these services. services is both common sense and good manage- ment. It needs to be recognized that performance The ministry or agency will likely need to help data obtained from M&E systems do not replace lower-level governments develop their capacity to the need for judgments by public officials. M&E operate their own M&E systems. The central min- information can only serve as one input, albeit istry or agency might need to provide guidance, a major input, for decision making. Many other guidelines, and �nancial support. The national factors will also need to be considered. agency will need to check the quality of the per- Implementing M&E successfully and usefully formance data from lower-level governments; this in a ministry or agency requires commitment and means providing training and technical assistance, leadership from high-level officials in these orga- but also periodically auditing at least a sample of nizations. And it takes time and special resources. the data and data collection procedures used by If the organization’s climate will not likely permit lower levels of government. meaningful use of M&E information, the effort will be a waste of money. However, if the climate is favorable, the ultimate gains should be consid- The added cost will depend considerably on the erable in improving the organization’s services to extent to which each agency program is already its citizens. collecting reasonably reliable performance data and if the ministry, agency, and program already have personnel that can assist in implementation. This brief note provides an overall picture of steps Many agency programs are likely to be already for implementing an M&E process in a ministry tracking a number of outcomes, which can be used or agency. Considerably more detail is provided as a starting point for the M&E system. in the following literature. The largest added costs will likely include the Gorgens, Marelize, and Jody Zall Kusek. 2009. Making costs for any new personnel needed (such as for Monitoring and Evaluation Systems Work. World analysis); training, including start-up, continuing, Bank. and replacement training; and for any new data Hatry, Harry P. 2006. Performance Measurement: Getting collection procedures, such as the cost of surveys. Results, Second Edition. Urban Institute. Household surveys, in particular, will probably Mackay, Keith. 2007. How to Build M&E Systems to need to be administered by an outside business, Support Better Government. World Bank. Zall Kusek, Jody, and Ray C. Rist. 2004. Ten Steps to Environmental Protection Agency, and has partici- a Results-Based Monitoring and Evaluation System. pated in projects in Albania, Colombia, Hungary, World Bank. Indonesia, Pakistan, and Thailand. Harry P. Hatry is a Distinguished Fellow and Director 1. The author thanks Markus Goldstein (Senior of the Public Management Program for the Urban Institute in Washington, DC. He has been a leader in Economist, Africa PREM), and these members of developing performance management and perfor- the Poverty Reduction and Equity Group: Helena mance management procedures for federal, state, Hwang (Consultant), Philipp Krause (Consul- and local public and private agencies. His book, tant), Gladys Lopez-Acevedo (Senior Economist), Performance Measurement: Getting Results, Second Keith Mackay (Consultant) and Jaime Saavedra Edition, is widely used and has been translated into (Acting Sector Director) for their comments. two other languages. He has provided assistance The views expressed in this note are those of the on M&E to the U.S. Departments of Education author. To access other notes in this series, visit and Justice, Health and Human Services and the www.worldbank.org/poverty/nutsandbolts. This note series is intended to summarize good practices and key policy �ndings on PREM-related topics. The views expressed in the notes are those of the authors and do not necessarily reflect those of the World Bank. PREMnotes are widely distributed to Bank staff and are also available on the PREM Web site (http://www. worldbank.org/prem). If you are interested in writing a PREMnote, email your idea to Madjiguene Seck at mseck@worldbank.org. For additional copies of this PREMnote please contact the PREM Advisory Service at x87736. This series is for both external and internal dissemination