A H a n d b o o k f o r D e v e l o p m e n t P r a c t i t i o n e r s Ten Steps to a Results- Based Monitoring and Evaluation System Jody Zall Kusek Ray C. Rist THE WORLD BANK A Handbook for Development Practitioners Ten Steps to a Results-Based Monitoring and Evaluation System A Handbook for Development Practitioners Ten Steps to a Results-Based Monitoring and Evaluation System Jody Zall Kusek Ray C. Rist THE WORLD BANK Washington, D.C. © 2004 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 Telephone 202-473-1000 Internet www.worldbank.org E-mail feedback@worldbank.org All rights reserved. 1 2 3 4 07 06 05 04 The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorse- ment or acceptance of such boundaries. Rights and Permissions The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. Library of Congress Cataloging-in-Publication Data Kusek, Jody Zall, 1952­ Ten steps to a results-based monitoring and evaluation system : a hand- book for development practitioners / Jody Zall Kusek and Ray C. Rist. p. cm. Includes bibliographical references and index. ISBN 0-8213-5823-5 1. Government productivity--Developing countries--Evaluation. 2. Performance standards--Developing countries--Evaluation. 3. Total quality management in government--Developing countries--Evaluation. 4. Public administration--Developing countries--Evaluation. I. Rist, Ray C. II. Title. JF1525.P67K87 2004 352.35--dc22 2004045527 Contents Preface xi About the Authors xiv Introduction Building a Results-Based Monitoring and Evaluation System 1 Part I New Challenges in Public Sector Management 2 International and External Initiatives and Forces for Change 3 National Poverty Reduction Strategy Approach 8 Internal Initiatives and Forces for Change 10 Part 2 Results-BasedM&E--APowerfulPublicManagementTool 11 Monitoring and Evaluation: What Is It All About? 12 Key Features of Traditional Implementation-Focused and Results- Based M&E Systems 15 Many Applications for Results-Based M&E 17 Political and Technical Challenges to Building a Results-Based M&E System 20 Introducing the 10-Step Model for Building a Results-Based M&E System 23 Where to Begin: Whole-of-Government, Enclave, or Mixed Approach 24 Part 3 M&E Experience in Developed and Developing Countries 27 M&E Experience in Developed and OECD Countries 27 Special M&E Challenges Facing Developing Countries 32 M&E Experience in Developing Countries 35 Chapter 1 Step 1: Conducting a Readiness Assessment 39 Part 1 Why Do a Readiness Assessment? 40 Part 2 The Readiness Assessment: Eight Key Questions 43 Part 3 Readiness Assessments in Developing Countries: Bangladesh, Egypt, and Romania 48 vi Contents Part 4 Lessons Learned 49 Chapter 2 Step 2: Agreeing on Outcomes to Monitor and Evaluate 56 The Importance of Outcomes 56 Issues to Consider in Choosing Outcomes to Monitor and Evaluate 57 The Importance of Building a Participatory and Consultative Process involving Main Stakeholders 58 The Overall Process of Setting and Agreeing upon Outcomes 59 Examples and Possible Approaches 61 Chapter 3 Step 3: Selecting Key Performance Indicators to Monitor Outcomes 65 Indicators Are Required for All Levels of Results-Based M&E Systems 66 Translating Outcomes into Outcome Indicators 66 The "CREAM" of Good Performance Indicators 68 The Use of Proxy Indicators 70 The Pros and Cons of Using Predesigned Indicators 72 Constructing Indicators 74 Setting Indicators: Experience in Developing Countries 75 Chapter 4 Step 4: Setting Baselines and Gathering Data on Indicators 80 Establishing Baseline Data on Indicators 81 Building Baseline Information 82 Identifying Data Sources for Indicators 83 Designing and Comparing Data Collection Methods 84 The Importance of Conducting Pilots 86 Data Collection: Two Developing Country Experiences 89 Chapter 5 Step 5: Planning for Improvement--Selecting Results Targets 90 Definition of Targets 90 Factors to Consider When Selecting Performance Indicator Targets 91 Examples of Targets Related to Development Issues 93 The Overall Performance-Based Framework 94 Chapter 6 Step 6: Monitoring for Results 96 Part 1 Key Types and Levels of Monitoring 98 Links between Implementation Monitoring and Results Monitoring 101 Part 2 Key Principles in Building a Monitoring System 103 Achieving Results through Partnership 105 Needs of Every Results-Based Monitoring System 106 The Data Quality Triangle: Reliability, Validity, and Timeliness 108 Contents vii Analyzing Performance Data 111 Pretesting Data Collection Instruments and Procedures 112 Chapter 7 Step 7: The "E" in M&E--Using Evaluation Information to Support a Results-Based Management System 113 Uses of Evaluation 115 The Timing of Evaluations 118 Types of Evaluations 121 Characteristics of Quality Evaluations 126 Examples of Evaluation at the Policy, Program, and Project Levels 128 Chapter 8 Step 8: Reporting the Findings 129 The Uses of Monitoring and Evaluation Findings 130 Know and Target the Audience 130 Presentation of Performance Data in Clear and Understandable Form 132 What Happens If the M&E System Produces Bad Performance News? 136 Chapter 9 Step 9: Using the Findings 138 Uses of Performance Findings 138 Additional Benefits of Using Findings: Feedback, Knowledge, and Learning 140 Strategies for Sharing Information 146 Chapter 10 Step 10: Sustaining the M&E System within the Organization 151 Six Critical Components of Sustaining Results-Based M&E Systems 152 The Importance of Incentives and Disincentives in Sustaining M&E Systems 155 Possible Problems in Sustaining Results-Based M&E Systems 155 Validating and Evaluating M&E Systems and Information 160 M&E: Stimulating Positive Cultural Change in Governments and Organizations 160 Last Reminders 160 Chapter 11 Making Results-Based M&E Work for You and Your Organization 162 Why Results-Based M&E? 162 How to Create Results-Based M&E Systems 165 Summing Up 170 viii Contents Annexes: Annex I: Assessing Performance-Based Monitoring and Evaluation Capacity: An Assessment Survey for Countries, Development Institutions, and Their Partners 174 Annex II: Readiness Assessment: Toward Results-Based Monitoring and Evaluation in Egypt 178 Annex III: Millennium Development Goals (MDGs): List of Goals and Targets 200 Annex IV: National Evaluation Policy for Sri Lanka: Sri Lanka Evaluation Association (SLEva) jointly with the Ministry of Policy Development and Implementation 204 Annex V: Andhra Pradesh (India) Performance Accountability Act 2003: (Draft Act) (APPAC Act of 2003) 211 Annex VI: Glossary: OECD Glossary of Key Terms in Evaluation and Results-Based Management (2002) 223 Notes 230 References 231 Useful Web Sites 235 Additional Reading 236 Index 239 Boxes i.i Millennium Development Goals 4 i.ii Example of Millennium Development Goal, Targets, and Indicators 5 i.iii Transparency International 6 i.iv The Power of Measuring Results 11 i.v Key Features of Implementation Monitoring versus Results Monitoring 17 i.vi Australia's Whole-of-Government Model 29 i.vii France: Lagging Behind but Now Speeding Ahead in Governmental Reform 30 i.viii Republic of Korea: Well on the Road to M&E 31 i.ix Malaysia: Outcome-Based Budgeting, Nation Building, and Global Competitiveness 36 i.x Uganda and Poverty Reduction--Impetus toward M&E 37 1.1 The Case of Bangladesh--Building from the Bottom Up 50 1.2 The Case of Egypt--Slow, Systematic Moves toward M&E 51 1.3 The Case of Romania--Some Opportunities to Move toward M&E 52 3.1 Indicator Dilemmas 71 3.2 The Africa Region's Core Welfare Indicators 76 3.3 Sri Lanka's National Evaluation Policy 77 3.4 Albania's Three-Year Action Plan 78 3.5 Program and Project Level Results Indicators: An Example from the Irrigation Sector 79 3.6 Outcome: Increased Participation of Farmers in Local Markets 79 4.1 Albania's Strategy for Strengthening Data Collection Capacity 88 Contents ix 4.2 Lebanon: Joining the IMF Data System 89 5.1 Examples of Development Targets 94 6.1 Results Monitoring in Mexico 101 6.2 Results Monitoring in Brazil 102 7.1 Evaluation Provides Information on Strategy, Operations, and Learning 117 9.1 Ten Uses of Results Findings 139 9.2 Using Performance Data to Track and Reduce Crime in New York City 141 9.3 U.S. Department of Labor--An Organization with a Mature, Functioning Results-Based M&E System 142 9.4 Signs of Improving Conditions for Evaluation-Based Learning in German Aid Agencies 144 9.5 Obstacles to Learning 145 9.6 Incentives for Learning, Knowledge Building, and Greater Use of Performance Findings 146 9.7 Active and Passive Approaches to Using Results Information 147 9.8 Canadian Government Performance Reports to Parliament 149 10.1 Citizen's Charter in the United Kingdom 155 10.2 U.S. Government Performance and Results Act of 1993 156 10.3 Checklist for Staff Incentives That Encourage Learning-Oriented, Participatory M&E 158 10.4 Checklist for Staff Disincentives That Hinder Learning-Oriented, Participatory M&E 158 10.5 An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity 161 Tables i.i Complementary Roles of Results-Based Monitoring and Evaluation 14 4.1 Building Baseline Information 82 4.2 Comparison of Major Data Collection Methods 87 8.1 Outcomes Reporting Format: Actual Outcomes versus Targets 133 8.2 Sample Table for Reporting Descriptive Data: Gender Differences in Voting 135 10.1 Evaluation Capacity Development and Institutionalization--Key Issues Addressed in Colombia, China, and Indonesia 157 Figures i.i Illustrative Logic Model for One National Development Goal 18 i.ii Ten Steps to Designing, Building, and Sustaining a Results-Based Monitoring and Evaluation System 25 1.1 Conducting a Readiness Assessment 39 2.1 Agreeing on Outcomes to Monitor and Evaluate 56 2.2 Developing Outcome Statements 60 2.3 Outcome Statements Derived from Identified Problems or Issues 62 2.4 How NOT to Construct Outcome Statements 63 2.5 Developing Outcomes for One Policy Area 64 x Contents 3.1 Selecting Key Indicators to Monitor Outcomes 65 3.2 Developing a Set of Outcome Indicators for a Policy Area 68 3.3 Checklist for Assessing Proposed Indicators 71 4.1 Baseline Data on Indicators--Where Are We Today? 80 4.2 Developing Baseline Data for One Policy Area 81 4.3 Data Collection Methods 85 5.1 Planning for Improvement--Selecting Results Targets 90 5.2 Identifying Desired Level of Results Requires Selecting Performance Targets 91 5.3 Developing Targets for One Policy Area 95 6.1 Monitoring for Results 96 6.2 Sample Gant Chart 97 6.3 Results-Based Monitoring 99 6.4 Examples of Results Monitoring 100 6.5 Links between Implementation Monitoring and Results Monitoring 103 6.6 Linking Implementation Monitoring to Results Monitoring 104 6.7 Achieving Results through Partnership 106 6.8 Every Monitoring System Needs Ownership, Management, Maintenance, and Credibility 107 6.9 Key Criteria for Collecting Quality Performance Data 109 6.10 The Data Quality Triangle: Reliability 109 6.11 The Data Quality Triangle: Validity 110 6.12 The Data Quality Triangle: Timeliness 110 6.13 Analyzing Results Data 111 7.1 The Role of Evaluations 113 7.2 Using Evaluation to Explain Performance Divergence 118 7.3 Using Evaluation to Determine the Impacts of Design and Implementation on Outcome 119 7.4 Seven Types of Evaluations 121 7.5 Characteristics of Quality Evaluations 126 7.6 Examples of Evaluation 128 8.1 Reporting Findings 129 8.2 Principles of Graphic Excellence and Sample Charts for Displaying Information 137 9.1 Using Findings 138 10.1 Sustaining the M&E System within the Organization 151 Preface An effective state is essential to achieving sustainable socioeconomic development. With the advent of globalization, there are growing pressures on governments and organizations around the world to be more responsive to the demands of internal and external stakeholders for good governance, accountability and transparency, greater devel- opment effectiveness, and delivery of tangible results. Governments, parliaments, citizens, the private sector, nongovernmental organiza- tions (NGOs), civil society, international organizations, and donors are among the stakeholders interested in better performance. As de- mands for greater accountability and real results have increased, there is an attendant need for enhanced results-based monitoring and evaluation of policies, programs, and projects. Monitoring and evaluation (M&E) is a powerful public manage- ment tool that can be used to improve the way governments and or- ganizations achieve results. Just as governments need financial, human resource, and accountability systems, governments also need good performance feedback systems. There has been an evolution in the field of monitoring and evalua- tion involving a movement away from traditional implementation- based approaches toward new results-based approaches. The latter help to answer the "so what" question. In other words, governments and organizations may successfully implement programs or policies, but have they produced the actual, intended results. Have govern- ments and organizations truly delivered on promises made to their stakeholders? For example, it is not enough to simply implement health programs and assume that successful implementation is equiv- alent to actual improvements in public health. One must also exam- ine outcomes and impacts. The introduction of a results-based M&E system takes decisionmakers one step further in assessing whether and how goals are being achieved over time. These systems help to answer the all important "so what" question, and respond to stake- holders' growing demands for results. xi xii Preface This handbook is primarily targeted toward officials who are faced with the challenge of managing for results. Developing coun- tries in particular have multiple obstacles to overcome in building M&E systems. However, as we shall see, results-based M&E systems are a continuous work in progress for both developed and develop- ing countries. As we have learned, when implemented properly these systems provide a continuous flow of information feedback into the system, which can help guide policymakers toward achieving the de- sired results. Seasoned program managers in developed countries and international organizations--where results-based M&E systems are now in place--are using this approach to gain insight into the per- formance of their respective organizations. This handbook can stand alone as a guide on how to design and construct a results-based M&E system in the public sector. It can also be used in conjunction with a workshop developed at the World Bank entitled "Designing and Building a Results-Based Monitoring and Evaluation System: A Tool for Public Sector Management." The goal of the handbook is to help prepare you to plan, design, and im- plement a results-based M&E system within your organization. In addition, the handbook will also demonstrate how an M&E system can be a valuable tool in supporting good public management. The focus of the handbook is on a comprehensive ten-step model that will help guide you through the process of designing and build- ing a results-based M&E system. These steps will begin with a "Readiness Assessment" and will take you through the design, man- agement, and, importantly, the sustainability of your M&E system. The handbook will describe these steps in detail, the tasks needed to complete them, and the tools available to help you along the way. Please also note the additional materials available in the annexes that can be used to enhance your understanding of the strategy de- scribed here for building your own results-based M&E system. We owe a special note of gratitude to the Policy and Operations Review Department of the Dutch Ministry of Foreign Affairs, specifi- cally to Rob D. van den Berg and Hans Slot. Through their financial support (via a Dutch Trust Fund at the World Bank) and their intel- lectual encouragement, they have been prime supporters of this ini- tiative. That this handbook has come to fruition is profoundly due to their consistency and vision. We also want to acknowledge with special thanks the contribution of Dr. Barbara Balaj to the preparation of this handbook. Her keen Preface xiii analytic insights, her thoughtful critiques, and her sustained support were invaluable. Her involvement significantly strengthened this handbook. We would also like to acknowledge the comments and critiques from the following colleagues here in the Bank, Osvaldo Feinstein and Laura Rawlings. We also want to thank Jonathan Breaul and Frans Leeuw for their constructive reviews as well. Their efforts are most appreciated. Building a results-based M&E system takes time. There will be many twists and turns along the road, but the journey and rewards are well worth it. Jody Zall Kusek Ray C. Rist Washington, D.C. About the Authors Jody Zall Kusek is the World Bank Africa Region Results Monitoring and Evaluation Coordinator. She advises on strategies to improve the capacity of M&E in both Bank and client organizations. Previously she was a Senior Evaluation Officer at the World Bank, implementing Bankwide improvement initiatives in the area of results-based monitoring and evaluations. Before joining the World Bank, Ms. Kusek was Director of Performance Planning for the U.S. Secretary of the Interior and Principal Management Advisor to the U.S. Secretary of Energy. Previous work also includes leading the Natural Resource Management Performance Review for former U.S. President Clinton. She has worked in Albania, Egypt, the Kyrgyz Republic, Mozambique, Romania, and Zambia to support the de- velopment of national monitoring and evaluation systems. She has recently published 10 articles in the area of poverty monitoring sys- tem development and management, and serves on the editorial board of a U.S. government knowledge and learning journal. Ray C. Rist is a Senior Evaluation Officer in the Operations Evaluation Department of the World Bank. His previous position in the Bank was as Evaluation Advisor and Head of the Evaluation and Scholarship Unit of the World Bank Institute. Prior to coming to the World Bank in 1996, his career included 15 years in the United States government with appointments in both the Executive and Legislative Branches. He served as a university professor with posi- tions at Johns Hopkins University, Cornell University, and George Washington University. Dr. Rist was the Senior Fulbright Fellow at the Max Planck Institute in Berlin, Germany, in 1976 and 1977. He has authored or edited 24 books, written more than 125 articles, and lectured in more than 60 countries. Dr. Rist serves on the edito- rial boards of nine professional journals and also serves as chair of an international working group that collaborates on research related to evaluation and governance. xiv Introduction Building a Results-Based Monitoring and Evaluation System While the role of the state has changed and evolved during recent his- "Good government is not a tory, it is now readily apparent that good governance is key to luxury--it is a vital neces- achieving sustainable socioeconomic development. States are being sity for development." challenged as never before by the demands of the global economy, new information and technology, and calls for greater participation (World Bank 1997, p. 15) and democracy. Governments and organizations all over the world are grappling with internal and external demands and pressures for improvements and reforms in public management. These demands come from a variety of sources including multilateral development institutions, donor governments, parliaments, the private sector, NGOs, citizens' groups and civil society, the media, and so forth. Whether it is calls for greater accountability and transparency, en- hanced effectiveness of development programs in exchange for for- eign aid, or real results of political promises made, governments and organizations must be increasingly responsive to internal and exter- nal stakeholders to demonstrate tangible results. "The clamor for greater government effectiveness has reached crisis proportions in many developing countries where the state has failed to deliver even such fundamental public goods as property rights, roads, and basic health and education" (World Bank 1997, p. 2). In short, govern- ment performance has now become a global phenomenon. Results-based monitoring and evaluation (M&E) is a powerful public management tool that can be used to help policymakers and decisionmakers track progress and demonstrate the impact of a given project, program, or policy. Results-based M&E differs from tradi- tional implementation-focused M&E in that it moves beyond an em- phasis on inputs and outputs to a greater focus on outcomes and im- pacts. Building and sustaining results-based M&E systems is not easy. It 1 2 Ten Steps to a Results-Based Monitoring and Evaluation System requires continuous commitment, time, effort, and resources--and champions--but it is doable. Once the system is built, the challenge is to sustain it. There are many political, organizational, and techni- cal challenges to overcome in building these systems--both for devel- oped and developing countries. Building and sustaining such systems is primarily a political process, and less so a technical one. There is no one correct way to build such systems, and many countries and organizations will be at different stages of development with respect to good public management practices in general, and M&E in partic- ular. It is important to recognize that results-based M&E systems are continuous works in progress. Developed countries, particularly those of the Organisation for European Co-operation and Development (OECD), have had as many as 20 or more years of experience in M&E, while many devel- oping countries are just beginning to use this key public management tool. The experiences of the developed countries are instructive, and can provide important lessons for developing countries. Developed countries have chosen a variety of starting points for implementing results-based M&E systems, including whole-of-government, en- clave, or mixed approaches--that may also be applicable to develop- ing countries. For their part, developing countries face a variety of unique challenges as they try to answer the "so what" question: What are the results and impacts of government actions? This introduction is divided into three parts. First, it focuses on the new challenges in public sector management, namely the many inter- nal and external pressures facing governments and organizations to manage for results. Second, it examines the use of M&E as a public management tool that can be utilized to track and demonstrate re- sults. Third, it documents the M&E experience in developed coun- tries, as well as the special challenges facing developing countries. PART 1 New Challenges in Public Sector Management There has been a global sea change in public sector management as a variety of internal and external forces have converged to make gov- ernments and organizations more accountable to their stakeholders. Governments are increasingly being called upon to demonstrate re- sults. Stakeholders are no longer solely interested in organizational activities and outputs; they are now more than ever interested in ac- Introduction: Building a Results-Based Monitoring and Evaluation System 3 tual outcomes. Have policies, programs, and projects led to the de- sired results and outcomes? How do we know we are on the right One public management track? How do we know if there are problems along the way? How lesson drawn from more can we correct them at any given point in time? How do we measure than 25 years of experi- progress? How can we tell success from failure? These are the kinds ence in OECD and devel- of concerns and questions being raised by internal and external oped countries is that stakeholders, and governments everywhere are struggling with ways building greater accounta- of addressing and answering them. bility within government will improve its overall International and External Initiatives and Forces for Change functioning. The same There are an increasing number of international initiatives and forces should also hold true for at work pushing and prodding governments in the direction of the developing world. adopting public management systems geared toward reform and, above all, results. These include: · Millennium Development Goals (MDGs) · Highly Indebted Poor Country (HIPC) Initiative · International Development Association (IDA) funding · World Trade Organization (WTO) membership · European Union (EU) enlargement and accession · European Union Structural Funds · Transparency International. The MDGs are among the most ambitious of global initiatives to adopt a results-based approach toward poverty reduction and im- provement in living standards. The eight comprehensive MDGs (box i.i) were adopted by 189 U.N. member countries and numerous inter- national organizations in 2000. They consist of a series of goals for the international community--involving both developed and devel- oping nations--to achieve by the year 2015.1 This new development agenda emphasizes the need to measure the results of aid financing. Are development initiatives making a differ- ence and having an impact? How will governments know whether they have made progress and achieved these goals? How will they be able to tell success from failure, or progress from setbacks? How will they identify obstacles and barriers? And at the most elementary level, do they even know their starting points and baselines in rela- tion to how far they must go to reach their goals? The MDGs contain some elements of a results-based M&E ap- proach. For example, the MDG targets have been translated into a set of indicators that can measure progress. Box i.ii contains an ex- 4 Ten Steps to a Results-Based Monitoring and Evaluation System Box i.i Millennium Development Goals 1. Eradicate extreme poverty and hunger 2. Achieve universal primary education 3. Promote gender equality and empower women 4. Reduce child mortality 5. Improve maternal health 6. Combat HIV/AIDS, malaria, and other diseases 7. Ensure environmental sustainability 8. Develop a global partnership for development. Source: United Nations ample of just one of the ways in which the goals have been articu- "The MDGs symbolize a lated into a series of targets and indicators. focus on results. . . . The More generally, the building and sustaining of comprehensive re- new development paradigm sults-based M&E systems at the country and donor levels will be key emphasizes results, partner- to measuring and monitoring achievement of the MDGs. ship, coordination, and ac- The 2002 Monterrey, Mexico, conference specifically addressed countability. . . . [It] com- means of achieving the MDGs. A new international consensus was bines a results-orientation; forged whereby developed countries would provide increased levels domestic ownership of im- of aid in conjunction with better governance, reform policies, and a proved policies; partner- greater focus on development effectiveness and results on the part of ships between govern- developing countries. ments, the private sector, The MDGs are also posing special challenges to the international and the civil society; and a evaluation community. It is becoming increasingly clear that a new long-term, holistic approach evaluation architecture is necessary. A foundation must be laid to that recognizes the interac- build results-based M&E systems beyond the country level by har- tion between development monizing and coordinating them internationally with U.N. agencies, sectors and themes." multilateral and bilateral donors, civil society, and the like. This will (Picciotto 2002, p. 3) be the future challenge in expanding M&E. Many countries, particularly the developing countries, must now vie to become a part of international initiatives, organizations, and blocs in order to reap the desired socioeconomic, political, and secu- rity benefits. Part of the bargain inevitably involves adhering to a set of specific requirements, conditions, and goals--including monitor- ing and evaluation. If these governments are going to become a part Introduction: Building a Results-Based Monitoring and Evaluation System 5 Box i.ii Example of Millennium Development Goal, Targets, and Indicators Goal: Eradicate extreme poverty and hunger Target l. Halve, between 1990 and 2015, the proportion of people whose income is less than US$1 a day Indicator 1. Proportion of population below US$1 per day Indicator 2. Poverty gap ratio (incidence × depth of poverty) Indicator 3. Share of poorest quintile in national consumption Target 2. Halve, between 1990 and 2015, the proportion of people who suffer from hunger Indicator 4. Prevalence of underweight children (under 5 years of age) Indicator 5. Proportion of population below minimum level of dietary energy consumption Source: United Nations 2003. of the global community, they must open themselves up to increased scrutiny and be more transparent and accountable to their stakehold- ers. In this context, they must learn to manage for results. Box i.iii describes the impact one external organization, Transparency Inter- national (TI), is having on the move toward accountability. The following are examples of the kinds of international initiatives and requirements set forth for joining international organizations and blocs--and for reaping the benefits of membership and inclu- sion. Together they have created a global force for public accounta- bility and proven results: · Highly Indebted Poor Country Initiative. In 1996, the World Bank and the International Monetary Fund (IMF) proposed the Highly Indebted Poor Country (HIPC) Initiative, the first com- prehensive approach to reduce the external debt of the world's poorest and most heavily-indebted countries. HIPC also aims at supporting poverty reduction, stimulating private sector­led growth and improvement in a country's social indicators. As a 6 Ten Steps to a Results-Based Monitoring and Evaluation System Box i.iii Transparency International "Transparency International is the only international organization exclusively devoted to curbing corruption" (TI 1997). Transparency International's (TI's) annual Corruption Perception Index--which ranks 102 countries by perceived levels of corruption among public officials--is cited by the world's media as the leading index in the field. TI's Bribe Payers Index ranks the leading exporting countries according to their propensity to bribe. TI is politically nonpartisan, and has chapters in 88 countries that carry out the anticorruption mission at the national level, helping to spread public awareness of corruption issues and the attendant detrimental development impact. "Corruption undermines good government, fundamentally distorts public policy, leads to the misallocation of resources, harms the private sector and private sector development and particularly hurts the poor" (TI 2002). TI is building coalitions with regional international institutions and actors to combat corruption. At the national level, TI is also working to build coalitions among all societal groups to strengthen governmental integrity systems. TI is also having an impact in monitoring performance at the multinational corporate level. "Trans- parency International's rise has coincided with many companies' discovering that they need to improve their image for being socially responsible in many countries. That has helped bolster the organization's fortunes and make it an important player in the global anti-corruption battle" (Crawford 2003, p. 1). With its broad international reach and media access, TI is yet another important global force for push- ing governments and multinational corporations to be more accountable, and to produce tangible results for their stakeholders. Source: TI 1997, 2002. condition for debt relief--and similar to the MDGs--recipient governments must be able to monitor, evaluate, and report on reform efforts and progress toward poverty reduction. For in- stance, Uganda made progress in M&E and qualified for en- hanced HIPC relief. In other cases, however, lack of capacity in building and maintaining results-based M&E systems has been a particular problem for participating HIPC countries such as Al- bania, Madagascar, and Tanzania. · International Development Association (IDA) funding. Under the IDA 13 replenishment negotiations--which resulted in the largest donor contribution ever (about US$23 billion)--39 Introduction: Building a Results-Based Monitoring and Evaluation System 7 donors based their support for 79 of the world's poorest coun- tries specifically on results. Explicit outcome indicators were for- mulated to track results toward goals, especially in health, edu- cation, and private sector development. IDA now has in place a Performance-Based Allocation system that has helped to better target donor resources to countries with good policies and institutions--in short, good governance. Tighter links are being achieved between performance and donor resource allocations. The assessments and resulting alloca- tions are increasingly being integrated in the country dialogue. With IDA 13, an initiative was also launched to put into place a comprehensive system to measure, monitor, and manage for development results. The system ties into current initiatives and is aligned with measurement systems established by IDA's bor- rowers under their National Poverty Reduction Strategy Papers, as well as their work toward achieving the MDGs. Efforts are also underway to ensure that this approach has wide acceptance and is coordinated with other actions being taken by the donor community (IDA 2002). · World Trade Organization membership. Other pressures come from the new rules of the game that have emerged with globali- zation, where demands for reduction of trade barriers have in- creased, and where financial capital and private sector interests demand a stable investment climate, the rule of law, and protec- tion of property and patents before investing in a given country. The WTO, successor to the General Agreement on Tariffs and Trade (GATT), is one such example. Created in 1995, the WTO facilitates the free flow of international trade. It has 147 mem- bers, and another 26 in the process of membership negotiations. Over three-quarters of WTO members are among the developing or least developed countries. Members must agree to comply with, and be monitored and evaluated against, a specific set of rules regarding reciprocity and equal treatment, transparency in trade and legal regimes, reduction of trade barriers, adoption of intellectual property rights legislation, and commitment to envi- ronmental protection. · European Union enlargement. The European Union (EU) has ex- perienced five separate enlargements during its history, growing from 6 to 25 member countries. The EU is and will be engaged in negotiations with additional countries on their accession ap- plications to join the EU. Aspiring countries must meet three 8 Ten Steps to a Results-Based Monitoring and Evaluation System basic criteria for accession: stable, democratic institutions and re- spect for human rights and minority protections; a functioning market economy capable of dealing with competitive pressures within the EU; and the ability to meet membership obligations associated with the political, economic, and monetary union. In this context, the EU monitors potential members' progress with respect to adopting, implementing, and applying EU legislation. National industries must also meet EU norms and standards. · EU Structural Funds. EU Structural Funds have been used to support and assist the socioeconomic development of the less- developed regions of EU member states. In an attempt to achieve greater socioeconomic cohesion within the EU, Struc- tural Funds have been used to redistribute funds to the poorer regions. Beneficiary regions have been required to establish a monitoring and evaluation process. As the EU enlarges, the Structural Funds will also be extended to include the lesser- developed regions of new members, thereby drawing them into the evaluation system as well. National Poverty Reduction Strategy Approach The Multilateral Development Banks (MDBs) have established strate- gies and approaches for sustainable development and poverty reduc- tion. These initiatives also involve setting goals, choosing indicators, and monitoring and evaluating for progress against these goals. · National Poverty Reduction Strategies. The HIPC initiative is also tied to National Poverty Reduction Strategies. In 1999, the international development community agreed that National Poverty Reduction Strategies should be the basis for concessional lending and debt relief. "Poverty Reduction Strategy Papers describe a country's macroeconomic, structural and social policies and programs to promote growth and reduce poverty, as well as associated exter- nal financing needs. PRSPs are prepared by governments through a participatory process involving civil society and development partners . . . " (World Bank 2003b). National Poverty Reduction Strategies must in turn be linked to agreed-upon development goals over a three year period-- with a policy matrix and attendant sets of measurable indicators, and a monitoring and evaluation system by which to measure Introduction: Building a Results-Based Monitoring and Evaluation System 9 progress. Specifically, "a PRSP will define medium and long-term goals for poverty reduction outcomes (monetary and nonmone- tary), establish indicators of progress, and set annual and medium-term targets. The indicators and targets must be appro- priate given the assessment of poverty and the institutional capacity to monitor. . . . a PRSP would [also] have an assessment of the country's monitoring and evaluation systems . . . " (World Bank 2003b). Thus, countries vying to become part of HIPC must commit to a process that involves accountability and transparency through monitoring, evaluation, and achievement of measurable results. · Comprehensive Development Framework. The Comprehensive Development Framework (CDF) consists of four basic principles: a long-term, holistic development framework; results orientation; country ownership; and country-led partnership. The CDF and National Poverty Reduction Strategies are mutually reinforcing; both also stress accountability for results. The adoption and application of the CDF--a systemic, long- term (generally 10 year) approach to development involving all stakeholders--has also resulted in pressures for the monitoring and evaluation of stakeholder participation and of economic development progress. The CDF includes in a country's national development strategy a clear delineation of medium- and long- term poverty reduction goals, with indicators to measure progress, thereby ensuring that policies are well designed, effec- tively implemented, and duly monitored. For example, stakeholders such as NGOs that have become involved in the process are looking for ways to monitor their own performance in terms of the National Poverty Reduction Strategy and the National Development Plan. The National Development Plan is now being implemented in a number of countries, and it is hoped that the approach will yield valuable information on set- ting baselines and measuring development outcomes. For ex- ample, the National Development Plan is a major force for devel- oping results-based M&E in the Kyrgyz Republic. A recent assessment of the CDF found that "Further research and exchange of experience among recipient countries are needed on how to build up country-owned monitoring and evaluation systems . . . " (World Bank 2003a, p. 4). 10 Ten Steps to a Results-Based Monitoring and Evaluation System Internal Initiatives and Forces for Change Governments are also facing increasing calls for reform from internal stakeholders, for example, to demonstrate accountability and trans- parency, devise fair and equitable public policies, and deliver tangible goods and services in a timely and efficient manner. Pressures may come from government officials, parliament, opposition parties, pro- gram managers and staff, citizens, businesses, NGOs, civil society, and the media. · Decentralization, deregulation, commercialization and privatiza- tion. The move toward various reforms, such as decentralization, deregulation, commercialization, or privatization, in many coun- tries has increased the need for monitoring and evaluation at re- gional and local levels of government. The need for monitoring also has increased as new nongovernmental service providers (such as NGOs, the private sector, and civil society groups) have begun taking over some of the public sector functions that were normally provided by governments in the past. As such initiatives are undertaken, there will be a continuing need to monitor and evaluate performance at different govern- mental and nongovernmental levels, as well as among new groups of stakeholders. For example, Colombia, Chile, and In- donesia are all undergoing fiscal decentralization, and are look- ing to build and extend evaluation responsibilities down to the local level. Although some governments may be diminishing their roles in providing public goods and services, they will still have a need to monitor and evaluate the impact of their policies and programs--regardless of who implements them. · Changes in government size and resources. There are many inter- nal pressures on governments to downsize and reform them- selves. Governments are experiencing budgetary constraints that force them to make difficult choices and tradeoffs in deciding on the best use of limited resources. The pressures to do more with less--and still demonstrate results--have grown. Governments are increasingly recognizing the need to build and sustain results- based M&E systems to demonstrate performance. There is a vast array of national, multilateral, and international forces, initiatives, and stakeholders calling on governments to be Introduction: Building a Results-Based Monitoring and Evaluation System 11 more accountable and transparent, and to demonstrate results. If de- veloping countries in particular are to join the globalization caravan and reap the benefits, they will need to meet specific requirements, standards, and goals. Results-based M&E systems can be a powerful public management instrument in helping them measure performance and track progress in achieving desired goals. PART 2 Results-Based M&E--A Powerful Public Management Tool This section examines the power of measuring performance (box i.iv), the history and definitions of M&E, the differences between traditional implementation-based M&E and the newer results-based M&E systems, and the complementary roles of monitoring and eval- uation. This section also explores the many applications of results- based M&E. The technical, organizational--and especially politi- cal--challenges involved in building a results-based M&E system are also addressed. Finally, the ten-step model to designing, building, and sustaining such systems, with some comments about how to approach ensuring sustainability of such systems in a given country, is introduced. There is tremendous power in measuring performance. The ancient Egyptians regularly monitored their country's outputs in grain and livestock production more than 5,000 years ago. In this sense, moni- toring and evaluation is certainly not a new phenomenon. Modern governments, too, have engaged in some form of traditional moni- toring and evaluation over the past decades. They have sought to Box i.iv The Power of Measuring Results · If you do not measure results, you cannot tell success from failure. · If you cannot see success, you cannot reward it. · If you cannot reward success, you are probably rewarding failure. · If you cannot see success, you cannot learn from it. · If you cannot recognize failure, you cannot correct it. · If you can demonstrate results, you can win public support. Source: Adapted from Osborne & Gaebler 1992. 12 Ten Steps to a Results-Based Monitoring and Evaluation System track over time their expenditures, revenues, staffing levels, resources, program and project activities, goods and services produced, and so forth. Governments have many different kinds of tracking systems as part of their management toolkits. Every government needs the three- legged stool of good human resource systems, financial systems, and accountability systems. But they also need good feedback systems. A results-based M&E system is essentially a special public management tool governments can use to measure and evaluate outcomes, and then feed this information back into the ongoing processes of govern- ing and decisionmaking. Monitoring and Evaluation: What Is It All About? Credible answers to the "so what" question address the accountabil- ity concerns of stakeholders, give public sector managers information on progress toward achieving stated targets and goals, and provide substantial evidence as the basis for any necessary mid-course correc- tions in policies, programs, or projects. Building an M&E system essentially adds that fourth leg to the governance chair. What typically has been missing from government systems has been the feedback component with respect to outcomes and consequences of governmental actions. This is why building an M&E system gives decisionmakers an additional public sector man- agement tool. The OECD (2002a) defines monitoring and evaluation as follows: Monitoring is a continuous function that uses the systematic col- lection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of ob- jectives and progress in the use of allocated funds (p. 27). Evaluation is the systematic and objective assessment of an on- going or completed project, program, or policy, including its design, implementation, and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. An evaluation should provide information that is credible and useful, enabling the in- corporation of lessons learned into the decisionmaking process of both recipients and donors (p. 21). Introduction: Building a Results-Based Monitoring and Evaluation System 13 (See annex 6 for a complete OECD glossary of key terms in evalua- tion and results-based management.) In juxtaposing these two definitions, it is immediately evident that they are distinct yet complementary. Monitoring gives information on where a policy, program, or project is at any given time (and over time) relative to respective targets and outcomes. It is descriptive in intent. Evaluation gives evidence of why targets and outcomes are or are not being achieved. It seeks to address issues of causality. Of par- ticular emphasis here is the expansion of the traditional M&E func- tion to focus explicitly on outcomes and impacts. Evaluation is a complement to monitoring in that when a monitor- ing system sends signals that the efforts are going off track (for ex- ample, that the target population is not making use of the services, that costs are accelerating, that there is real resistance to adopting an innovation, and so forth), then good evaluative information can help clarify the realities and trends noted with the monitoring system. For example, "If annual performance information is presented by itself (in isolation) without the context and benefit of program evaluation, there is a danger of program managers, legislators . . . and others drawing incorrect conclusions regarding the cause of improvements or declines in certain measures . . . Simply looking at trend data usu- ally cannot tell us how effective our government program interven- tions were" (ChannahSorah 2003, p. 7). We stress the need for good evaluative information throughout the life cycle of an initiative--not just at the end--to try and determine causality. Table i.i highlights the different--yet complementary--roles that monitoring and evaluation play in M&E systems. Monitoring can be done at the project, program, or policy levels. For example, in looking at infant health, one could monitor the proj- ect level by monitoring the awareness of good prenatal care in six targeted villages. At the program level, one could monitor to ensure that information on prenatal care is being targeted to pregnant women in a whole region of the country. At the policy monitoring level, the concern could be to monitor the overall infant morbidity and mortality rates for that same region. Evaluation, like monitoring, may be conducted at the project, program, or policy level. To take an example of privatizing water systems, a project evaluation might involve the assessment of the improvement in water fee collection rates in two provinces. At the program level, one might consider assessing the fiscal management 14 Ten Steps to a Results-Based Monitoring and Evaluation System Table i.i Complementary Roles of Results-Based Monitoring and Evaluation Monitoring Evaluation · Clarifies program · Analyzes why intended objectives results were or were not achieved · Links activities and their · Assesses specific causal resources to objectives contributions of activities to results · Translates objectives into · Examines implementation performance indicators process and sets targets · Routinely collects data on · Explores unintended these indicators, results compares actual results with targets · Reports progress to · Provides lessons, high- managers and alerts lights significant accom- them to problems plishment or program potential, and offers recommendations for improvement of the government's systems, while at the policy level, one might eval- uate different model approaches to privatizing public water supplies. When we refer to evaluation in the context of an M&E system, we are not solely referring to the classical approach of determining attri- bution as embodied in the after-the-fact assessment of projects, pro- grams, or policies. Impact evaluations do (or at least try to) address attribution. But we are viewing evaluation in a much broader context as a continuously available mode of analysis that helps program managers gain a better understanding of all aspects of their work-- from design through implementation and on to completion and sub- sequent consequences. We will also discuss later in this handbook the notion that what managers increasingly need are streams of evalua- tion information, not additional discrete and episodic evaluation studies. Introduction: Building a Results-Based Monitoring and Evaluation System 15 Evaluation has also been used for different purposes over the years. In the OECD countries, for example, early evaluations in the 1960s and 1970s studied ways of improving social programs. Later in the 1980s and 1990s, governments used evaluation to conduct budgetary management, for example, by examining ways to reduce expenditures and cut public programs. As noted earlier, efforts to de- velop M&E systems have spread to developing countries--many hav- ing been driven by the desire to meet specific donor requirements, in- ternational development goals, or, in some cases, both external and internal social and economic pressures. Again, evaluation can be defined as an assessment, as systematic and objective as possible, of a planned, ongoing, or completed inter- vention. The aim is to determine the relevance of objectives, effi- ciency, effectiveness, impact, and sustainability so as to incorporate lessons learned into the decisionmaking process. Specifically, this kind of evaluation addresses: "why" questions, that is, what caused the changes being monitored; "how" questions, or what was the se- quence or process that led to successful (or unsuccessful) outcomes; and "compliance and accountability" questions, that is, did the promised activities actually take place and as planned? Key Features of Traditional Implementation-Focused and Results-Based M&E Systems Traditional implementation-focused M&E systems are designed to address compliance--the "did they do it" question. Did they mobi- lize the needed inputs? Did they undertake and complete the agreed activities? Did they deliver the intended outputs (the products or services to be produced)? The implementation approach focuses on monitoring and assessing how well a project, program, or policy is being executed, and it often links the implementation to a particular unit of responsibility. However, this approach does not provide poli- cymakers, managers, and stakeholders with an understanding of the success or failure of that project, program, or policy. Results-based M&E systems are designed to address the "so what" question. So what about the fact that outputs have been gen- erated? So what that activities have taken place? So what that the outputs from these activities have been counted? A results-based sys- tem provides feedback on the actual outcomes and goals of govern- ment actions. Results-based systems help answer the following questions: 16 Ten Steps to a Results-Based Monitoring and Evaluation System · What are the goals of the organization? · Are they being achieved? · How can achievement be proven? Box i.v illustrates some of the key differences between traditional implementation-based M&E systems and results-based M&E systems. Results-based monitoring is a continuous process of collecting and analyzing information to compare how well a project, program, or policy is being implemented against expected results. Figure i.i illustrates the manner in which the monitoring and eval- uation of national development goals will have to include not only the traditional implementation focus, but also a results focus. It also shows how results-based systems build upon and add to traditional implementation-focused systems. We would note in figure i.i that by leaving the generation of out- puts as an implementation effort rather than as a result, we are at some variance from the OECD glossary, which defines results as in- cluding outputs together with outcomes and impacts. We do this to stress the focus on answering the "so what" question. Building a school, paving a road, or training rural clinic workers does not, in our view, answer the "so what" question. These are outputs--and now one goes on to say "so what." What are the results of having this school building, this paved road, or these trained clinic workers? As can be seen in figure i.i, monitoring progress toward national goals requires that information be derived in the logic model from all results levels, at different time frames, and for different stakeholder needs. A common strategy is to measure outputs (number of health professionals trained) but not improvements in performance (im- proved use of oral rehydration therapy [ORT] for managing child- hood diarrhea). Improved institutional performance is assumed, but seldom documented. Without measured results, there is no way to document whether the effort is actually achieving the expected out- comes (improved use of ORT), and ultimately the associated national goal (reduction in child mortality). So what does this mean in a governmental results-based M&E context? As governments seek to align the expenditure framework with policy outcomes, measuring the organization's performance in support of achieving outcomes is important. The efficiency of service delivery, the quality of program and policy implementation, and the effective management of resources are just a few examples. In the Introduction: Building a Results-Based Monitoring and Evaluation System 17 Box i.v Key Features of Implementation Monitoring versus Results Monitoring Elements of Implementation Monitoring (traditionally used for projects) · Description of the problem or situation before the intervention · Benchmarks for activities and immediate outputs · Data collection on inputs, activities, and immediate outputs · Systematic reporting on provision of inputs · Systematic reporting on production of outputs · Directly linked to a discrete intervention (or series of interventions) · Designed to provide information on administrative, implementation, and management issues as opposed to broader development effectiveness issues. Elements of Results Monitoring (used for a range of interventions and strategies) · Baseline data to describe the problem or situation before the intervention · Indicators for outcomes · Data collection on outputs and how and whether they contribute toward achievement of outcomes · More focus on perceptions of change among stakeholders · Systemic reporting with more qualitative and quantitative information on the progress toward outcomes · Done in conjunction with strategic partners · Captures information on success or failure of partnership strategy in achieving desired outcomes. Source: Adapted from Fukuda-Parr, Lopes, and Malik 2002, p. 11. Philippines, for instance, the government is at the early stages of defining organizational level indicators for major outcomes against which expenditure decisions can be made (World Bank 2001e). Many Applications for Results-Based M&E There are many and growing applications for results-based M&E. As the needs for accountability and demonstrable results have grown, so have the uses and applications for results-based M&E systems. Project, Program, and Policy Applications Results-based M&E sys- tems have been successfully designed and used to monitor and evalu- ate at all levels--project, program, and policy. Information and data 18 Ten Steps to a Results-Based Monitoring and Evaluation System Figure i.i Illustrative Logic Model for One National Development Goal Reduce mortality rates for children Goal under 5 years old Results Improved use of ORT Outcome for managing childhood diarrhea · 15 media campaigns completed Outputs · 100 health professionals trained · Increased maternal knowledge of ORT services · Increased access to ORT Activities · Launch media campaign to educate mothers · Train health professionals in ORT Implementation Inputs · Trainers · ORT supplies · Funds · Participants Source: Binnendijk 2000. Note: ORT = Oral Rehydration Therapy. Introduction: Building a Results-Based Monitoring and Evaluation System 19 can be collected and analyzed at any and all levels to provide feed- back at many points in time. In this way, the information can be used to better inform key decisionmakers, the general public, and other stakeholders. Monitoring and evaluation can and should be evident throughout the life cycle of a project, program, or policy, as well as after comple- tion. M&E--with its continuing streams of data and feedback-- has added value at every stage from design through implementation and impact. "The specific information will also be different at each level, the complexity of collecting data will be different, the political sensitivity on collecting the data may change, and the uses of the in- formation may change from one level to another" (Kusek and Rist 2001, p. 17). Internal and External Applications M&E can also be conducted at local, regional, and national levels of government. So whether one thinks of M&E in relation to levels of administrative complexity (project to program to policy) or geographically, the applications are evident--though they need not be identical. Again, the specific indi- cators may necessarily be different, as the stakeholders' needs for information will also be different for each level of government. It should also be noted that a functioning M&E system provides a continuous flow of information that is useful both internally and ex- ternally. The internal uses come into play as the information from the M&E system is used as a crucial management tool for the public sec- tor manager in achieving results and meeting specific targets. Infor- mation on progress, problems, and performance are all key to a pub- lic manager striving to achieve results. Likewise, the information from an M&E system is important to those outside the public sector who are expecting results, wanting to see demonstrable impacts from government action (and tax monies), and hoping to build trust in a government that is striving to better the life of its citizens. Fundamentally, the M&E system aids in thinking about and clari- fying goals and objectives. Governments and stakeholders can also use M&E systems for formulating and justifying budgetary requests. In contrast to the earlier implementation-based approach, results- based M&E focuses attention on achieving outcomes important to the organization and its internal and external stakeholders. M&E systems can help identify potentially promising programs or practices. They can also identify unintended--but perhaps useful-- 20 Ten Steps to a Results-Based Monitoring and Evaluation System project, program, and policy results. Conversely, M&E systems can help managers identify program weaknesses and take action to cor- rect them. An M&E strategy can be used to diminish fear within or- ganizations and governments, and can instead devise ways of instill- ing an open atmosphere in which people can learn from mistakes, make improvements, and create knowledge along the way. Knowledge Capital Good M&E systems are also a source of knowl- edge capital. They enable governments and organizations to develop a knowledge base of the types of projects, programs, and policies that are successful, and, more generally, what works, what does not, and why. M&E systems can also provide continuous feedback in the management process of monitoring and evaluating progress toward a given goal. In this context, they promote organizational learning. Broad public access to information derived from results-based M&E systems is also important in aiding economic development both within and between countries. "Access to information is an es- sential component of a successful development strategy. If we are se- rious about reducing global poverty, we must liberate the access to information and improve its quality" (Stiglitz and Islam 2003, p. 10). Transparency and Accountability M&E systems can also aid in pro- moting greater transparency and accountability within organizations and governments. Beneficial spillover effects may also occur from shining a light on results. External and internal stakeholders will have a clearer sense of the status of projects, programs, and policies. The ability to demonstrate positive results can also help garner greater political and popular support. There are organizational and political costs and risks associated with implementing results-based M&E systems. However, there are also crucial costs and risks involved in not implementing such systems. Political and Technical Challenges to Building a Results-Based M&E System There are a variety of political and technical challenges involved in building results-based systems. The political are often the most diffi- cult to overcome. The Political Side of M&E Implementing results-based M&E systems poses many political challenges in OECD and developing countries alike. Above all, it takes strong and consistent political leadership Introduction: Building a Results-Based Monitoring and Evaluation System 21 and will--usually in the form of a political champion--to institute Many organizations would such a system. Bringing results-based information into the public prefer to operate in the arena can change the dynamics of institutional relations, budgeting shadows. They do not and resource allocations, personal political agendas, and public per- want to publish data about ceptions of governmental effectiveness. Strong, vested interests may their performance and out- also perceive themselves to be under attack. There may be counter- comes. Instituting a results- reformers within and outside the government who actively oppose based M&E system sheds such efforts. Thus, the role of a political champion is key to ensuring light on issues of organiza- the institutionalization and sustainability of results-based M&E tional performance. Not all systems. stakeholders will be pleased Results-based M&E systems are essential components of the gov- to have such public expo- ernance structure--and are thus fundamentally related to the politi- sure. This is just one of the cal and power systems of government. M&E systems provide critical ways in which M&E sys- information and empower policymakers to make better-informed tems pose a political--more decisions. At the same time, providing such information may lessen than a technical--challenge. or otherwise constrain the number of options available to politi- cians--leaving them less room to maneuver in their policies. In democracies, information on project, program, and policy re- sults is increasingly essential and is expected in the normal course of government operations. It is assumed that such information can help and guide policymaking. However, M&E systems may pose special challenges for countries that have been previously ruled by central- ized, authoritarian political regimes. Instituting M&E systems that will highlight outcomes--both successes and failures--and provide greater transparency and accountability may be especially challeng- ing and even alien to such countries. It may require a longer time for the political class, citizenry, and culture to adapt and change. Finally, one cannot build strong economies on weak governments. By comparison with the Results-based M&E systems can help strengthen governments by re- politics of instituting re- inforcing the emphasis on demonstrable outcomes. Getting a better sults-based M&E systems, handle on the workings and outcomes of economic and governmen- technical issues are rela- tal programs and policies can contribute to poverty reduction, higher tively less complex to ad- economic growth, and the achievement of a wide range of develop- dress and solve. ment goals. The Technical Side of M&E--Building Institutional Capacity Designing and building a reporting system that can produce trust- worthy, timely, and relevant information on the performance of government projects, programs, and policies requires experience, skill, and real institutional capacity. This capacity for a results-based 22 Ten Steps to a Results-Based Monitoring and Evaluation System reporting system has to include, at a minimum, the ability to suc- cessfully construct indicators; the means to collect, aggregate, ana- lyze, and report on the performance data in relation to the indica- tors and their baselines; and managers with the skill and understand- ing to know what to do with the information once it arrives. Building such capacity in governments for these systems is a long- term effort. Some developing countries currently lack the basic capacity to suc- cessfully measure inputs, activities, and outputs. But all countries will eventually need to be able to technically monitor and track at each level of the results-based M&E system--at the input, activity, output (implementation), outcome, and impact (goal) levels. Statistical capacity is an essential component of building results- based M&E systems. Information and data should be valid, verifi- able, transparent, and widely available to the government and inter- ested stakeholders--including the general public. This may be difficult for some governments that would prefer not to disclose and share data for political reasons or to hide corruption. Technically trained staff and managers, and at least basic informa- tion technology, are also a must. In some cases, donor-supported technical assistance and training will first be necessary for the coun- try to produce a minimum of information and data, and start to build an M&E system. For example, a recent assessment found that capacity building for key national officials in results-based M&E and performance-based budgeting will be needed in the Arab Republic of Egypt (World Bank 2001c). In the case of Colombia, government of- ficials have commissioned an external evaluation of major projects while simultaneously building internal evaluation capacity. Sometimes a great deal of data are collected in a country, but there may not be much understanding of how to use the data. Collecting and dumping large amounts of data on managers is not helpful. Pro- viding mounds of data and no analysis will not generate the informa- tion needed to improve programs. How much information and data are enough? Obviously, decision- makers seldom have all the information they need when they need it. This is a common dilemma with respect to managing in any organiza- tion. Even without perfect data, though, if the M&E system can pro- vide some analytic feedback, it will help policymakers make more well-informed decisions. Introduction: Building a Results-Based Monitoring and Evaluation System 23 Introducing the 10-Step Model for Building a Results-Based M&E System Although experts vary on the specific sequence of steps in building a results-based M&E system, all agree on the overall intent. For ex- ample, different experts propose four- or seven-step models. Regard- less of the number of steps, the essential actions involved in building an M&E system are to: · Formulate outcomes and goals · Select outcome indicators to monitor · Gather baseline information on the current condition · Set specific targets to reach and dates for reaching them · Regularly collect data to assess whether the targets are being met · Analyze and report the results. Given the agreement on what a good system should contain, why are these systems not part of the normal business practices of govern- ment agencies, stakeholders, lenders, and borrowers? One evident reason is that those designing M&E systems often miss the complexi- ties and subtleties of the country, government, or sector context. Moreover, the needs of end users are often only vaguely understood by those ready to start the M&E building process. Too little empha- sis is placed on organizational, political, and cultural factors. In this context, the 10-step model presented here (Figure i.ii) differs from others because it provides extensive details on how to build, maintain--and perhaps most importantly--sustain a results- based M&E system. It also differs from other approaches in that it contains a unique readiness assessment. Such an assessment must be conducted before the actual establishment of a system. The readiness assessment is, in essence, the foundation of the M&E system. Just as a building must begin with a foundation, constructing an M&E sys- tem must begin with the foundation of a readiness assessment. With- out an understanding of the foundation, moving forward may be fraught with difficulties and, ultimately, failure. It is Step 1. Throughout, the model highlights the political, participatory, and partnership processes involved in building and sustaining M&E sys- tems, that is, the need for key internal and external stakeholders to be consulted and engaged in setting outcomes, indicators, targets, and so forth. Step 2 of the model involves choosing outcomes to monitor and evaluate. Outcomes show the road ahead. 24 Ten Steps to a Results-Based Monitoring and Evaluation System Step 3 involves setting key performance indicators to monitor progress with respect to inputs, activities, outputs, outcomes, and im- pacts. Indicators can provide continuous feedback and a wealth of performance information. There are various guidelines for choosing indicators that can aid in the process. Ultimately, constructing good indicators will be an iterative process. Step 4 of the model relates to establishing performance baselines-- qualitative or quantitative--that can be used at the beginning of the monitoring period. The performance baselines establish a starting point from which to later monitor and evaluate results. Step 5 builds on the previous steps and involves the selection of results targets, that is, interim steps on the way to a longer-term outcome. Targets can be selected by examining baseline indicator levels and desired levels of improvement. Monitoring for results, Step 6 of the model, includes both imple- mentation and results monitoring. Monitoring for results entails collecting quality performance data, for which guidelines are given. Step 7 deals with the uses, types, and timing of evaluation. Reporting findings, Step 8, looks at ways of analyzing and report- ing data to help decisionmakers make the necessary improvements in projects, policies, and programs. Step 9, using findings, is also impor- tant in generating and sharing knowledge and learning within gov- ernments and organizations. Finally, Step 10 covers the challenges in sustaining results-based M&E systems including demand, clear roles and responsibilities, trustworthy and credible information, accountability, capacity, and appropriate incentives. The 10-step system can be used for projects, programs, and poli- cies. Though visually it appears as a linear process, in reality it is not. One will inevitably move back and forth along the steps, or work on several simultaneously. The use of such results-based M&E systems can help bring about major cultural changes in the ways that organizations and govern- ments operate. When built and sustained properly, such systems can lead to greater accountability and transparency, improved perform- ance, and generation of knowledge. Where to Begin: Whole-of-Government, Enclave, or Mixed Approach Governments around the world differ in their approaches to adopt- ing results-based M&E systems. There are essentially three ap- Introduction: Building a Results-Based Monitoring and Evaluation System 25 Figure i.ii Ten Steps to Designing, Building, and Sustaining a Results-Based Monitoring and Evaluation System Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 6 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization proaches. The first is the whole-of-government approach that was adopted in some of the early M&E pioneer countries. The whole-of- government approach involves a broad, comprehensive establishment of M&E across the government. With the adoption of the MDGs, many developing countries are looking to design and implement comprehensive results-based M&E systems across many sectors and policies. Also, with the growing em- phasis on results in international aid lending, more donor govern- ments and institutions will likely provide support to developing countries to build broad M&E systems. There are trends among some donor agencies and governments to perform joint evaluations involving the recipient country as an active participant. Often, different ministries are at different stages in their ability to take on the establishment of an M&E system. The whole-of-govern- ment strategy may not be able to move all ministries in tandem; there may be a need for sequencing among ministries in developing these systems. Many times innovations at one level will filter horizontally and vertically to other levels in the government. Thus, the second approach is a more limited or enclave-focused one. Many countries--especially developing countries--may not yet be in a position to adopt such sweeping change in a comprehensive fashion. Other, more targeted approaches are available, such as be- 26 Ten Steps to a Results-Based Monitoring and Evaluation System ginning with the local, state, or regional governmental levels, or pi- loting M&E systems in a few key ministries or agencies. Interestingly, some countries, such as Ireland, have adopted a third, blended approach to M&E. While some areas are comprehen- sively monitored and evaluated (projects financed by the EU Struc- tural Funds, for example), other areas receive more sporadic atten- tion. The government of Ireland has moved in the direction of a more comprehensive evaluation approach with respect to government ex- penditure programs (Lee 1999). The blended approach may also be a plausible alternative for some developing countries. Piloting of M&E systems is often recommended, regardless of which approach is adopted. The best strategy to introduce an M&E system into a country is to first test a program in two or more pilot ministries. Albania, for example, is aligning a results-based M&E program with a newly implemented, medium-term expenditure framework, and pilot testing the effort in four key ministries. Egypt has selected six performance pilots to explore how performance- oriented budgeting could work before applying the approach to the government as a whole. Yet a third strategy for applying a results-oriented program is a focus on a particular customer group. The government of Egypt wanted to improve its programs and services to advance women's is- sues. Each line ministry was expected to identify its current programs related to gender issues and assess the performance of the programs. In addition, the National Council for Women, a recently established government organization aimed at improving government support to women, was to identify a set of key performance indicators that the government could then track and monitor to achieve the established gender-related goals. It is the responsibility of the related ministries to track and monitor indicators for programs within their ministerial control, and to closely monitor and evaluate related government pro- grams to achieve results (World Bank 2001c). There is power in measuring performance. Results-based M&E systems are a powerful public management tool in helping govern- ments and organizations demonstrate impacts and outcomes to their respective stakeholders, and to gain public support. Results-based systems are similar to traditional M&E systems, but move beyond them in their focus on outcomes and impacts--rather than simply ending with a focus on implementation, that is, inputs, activities, and outputs. Introduction: Building a Results-Based Monitoring and Evaluation System 27 In sum, these systems have many applications, and can be used at the project, program, or policy level. There are many political, insti- tutional, and technical challenges in building results-based M&E sys- tems. Furthermore, countries should choose whether to adopt a whole-of-government approach in instituting such systems, or to begin by implementing an enclave approach at only one level in a government, or within a single ministry or small cluster of ministries. Experiences differ between developed and developing countries in how they have chosen to approach the design and construction of results-based M&E systems. PART 3 M&E Experience in Developed and Developing Countries This section provides some background information on experiences with results-based M&E systems in developed and developing coun- tries. There is no one correct way to go about building such systems. Different countries--developed and developing alike--will be at dif- ferent stages with respect to constructing M&E systems. Within countries, different ministries or levels of government may be at dif- ferent stages of development in their M&E capacity. We will look at some of the special challenges facing developing countries as they try to build, operate, and sustain results-based M&E systems. M&E Experience in Developed and OECD Countries A large majority of the 30 OECD countries now have results-based Building an effective M&E M&E systems. Arriving there was neither an easy nor a linear system is easier said than process for them. They differ--often substantially--in their paths, done. There are a number approach, style, and level of development. According to a recent sur- of systems that function vey, Australia, Canada, the Netherlands, Sweden, and the United well in developed countries, States have the highest evaluation culture rankings among OECD and fewer in developing countries (Furubo, Rist, and Sandahl 2002). countries. It is not that The OECD countries have developed evaluation cultures and governments are not try- M&E systems in response to varying degrees of internal and external ing--many of them are. pressures. For example, France, Germany, and the Netherlands devel- But creating such a system oped such a culture in response to both strong internal and external takes time, resources, and a (mostly EU-related) pressures, while countries such as Australia, stable political environ- Canada, the Republic of Korea, and the United States were moti- ment--and strong champi- vated mostly by strong internal pressures. ons who do not become Interestingly, the pioneering OECD countries were motivated to faint of heart. 28 Ten Steps to a Results-Based Monitoring and Evaluation System adopt evaluation cultures mostly because of strong internal pressures. These countries were also instrumental in spreading the evaluation culture to other countries by disseminating evaluation ideas and in- formation, and launching evaluation organizations, training insti- tutes, networks, and consulting firms. By contrast, many of the latecomer countries (for example, Italy, Ireland, and Spain) tended to respond to evaluation issues primarily because of strong external pressures. They were also heavily influ- enced by the evaluation culture of the pioneers, as well as the evalua- tion culture that has taken root in the international organizations with which their countries interact. Boxes i.vi, i.vii, and i.viii give a brief overview of results-based M&E experiences in three OECD countries--Australia, France, and Korea. The motivations, approaches, and strategies differed in each case. Important conclusions and lessons can be drawn from these experiences. Indications of Progress to Date in OECD Countries A recent OECD survey provides a useful overview of the extent to which a results-based focus has permeated and taken root in OECD country budgetary and management systems and practices. For example, "most governments today include performance informa- tion in their budget documentation and that information is subject to some form of audit in half of the countries. Though the current debate in the international public management and budgeting community on the distinction between outcomes and outputs is relatively new, the distinction between the two categories of results is used in most or all organizations in 11 out of 27 countries" (OECD 2002b, p. 12). While substantial progress has been made in OECD countries on a number of fronts, there is still room for improvement. The OECD survey found "Only a limited number of countries link performance targets to expenditures for all government programs though around half of them have established links for some of their programs. A limited number of countries use performance targets without any linking to expenditure at all" (OECD 2002b, p. 12). Another weak- ness in OECD countries is that only "half of the countries reported that performance information is used for allocation purposes during the budget procedure but also that the use is confined to allocation within ministries and programs" (OECD 2002b, p. 12). Thus, while Introduction: Building a Results-Based Monitoring and Evaluation System 29 Box i.vi Australia's Whole-of-Government Model Australia was one of the early pioneers in developing M&E systems, starting in 1987. The country had a number of intrinsic advantages conducive to building a sound evaluative culture and structure: · Strong human, institutional, and management capacity in the public sector · Public service known for integrity, honesty, and professionalism · Well-developed financial, budgetary, and accounting systems · A tradition of accountability and transparency · Credible, legitimate political leaders. A variety of factors contributed to Australia's success in building strong M&E systems. Initially, budget- ary constraints prompted the government to look at ways of achieving better value for money. Australia also had two important institutional champions for evaluation--the Department of Finance and the Aus- tralian National Audit Office. Australia chose to adopt a whole-of-government strategy. Such a strategy aims to bring all ministries on board--both the leading and the reluctant. The effort also had the support of cabinet members and key ministers who placed importance on using evaluation findings to better inform decisionmaking. Australia's evaluation system evolved from one of tight, central controls imposed by the Department of Finance to a more voluntary and devolutionary principles-based approach. The latter approach has helped to increase evaluation commitment and ownership at the program level. Today, monitoring and evaluation is left up to the individual departments and agencies. The formal M&E requirements have been relaxed considerably, and departments conduct M&E based on their own priorities. At the same time, departments are still required to report performance information in budget documents, and to report evaluation findings where available. Additionally, some evaluations continue to be mandated by the cabinet. The larger governmental departments are particularly active in commissioning formal evaluations and using the findings. Source: Mackay 2002. progress has been made in instituting results-based M&E systems and procedures in many OECD countries, much remains to be done. Conclusions and Lessons from OECD Countries A number of factors contributed to the adoption of an evaluation culture in the pioneer- ing countries in particular. Many of the earliest adopters of M&E systems were predisposed to do so because they had democratic 30 Ten Steps to a Results-Based Monitoring and Evaluation System Box i.vii France: Lagging Behind but Now Speeding Ahead in Governmental Reform In contrast to other OECD countries, France was among the group that was slowest to move toward a results-based M&E system. Indeed, France even lagged behind many transition and developing economies. Various incremental reform efforts were attempted during the late 1980s and throughout the 1990s. However, in 2001 the French government passed sweeping legislation--replacing the 1959 financial constitutional bylaw eliminating line-item budgeting, and instituting a new program approach. The new constitutional bylaw, which will be phased in over a five-year period (2001­2006), has two primary aims: reform the public management framework to make it results and performance-oriented; and strengthen parliamentary supervision. As former Prime Minister Lionel Jospin noted: "The budget's presentation in the form of programs grouping together expenditure by major public policy should give both members of Parliament and citizens a clear picture of the government's priorities and the cost and results of its action." Approximately 100 to 150 programs were identified, and financial resources were budgeted against them. Every program budget that is submitted to parliament must have a statement of precise objectives and performance indicators. Public managers have greater freedom and autonomy with respect to the allo- cation of resources, but in return are held more accountable for results. Thus, the new budget process is completely results driven. Future budget bills will include annual performance plans, detailing the expected versus actual results for each program. Annual performance reports are also included in budgetary reviews. Consequently, members of parliament have the ability to evaluate the performance of these governmental programs. In line with the earlier observations about the political nature of M&E, this reform initiative altered some of the political and institutional relationships within the French government. In this context, parlia- ment has been given increased budgetary powers. "Article 40 of the Constitution previously prohibited members of Parliament from tabling amendments that would increase spending and reduce revenue. They will now be able to change the distribution of appropriations among programs in a give mission." Parlia- ment is able to vote on revenue estimates, appropriations for each mission, the limits on the number of state jobs created, and special accounts and specific budgets. In addition, the parliamentary finance committees have monitoring and supervisory responsibilities regarding the budget. Source: Republique Française 2001. Introduction: Building a Results-Based Monitoring and Evaluation System 31 Box i.viii Republic of Korea: Well on the Road to M&E In terms of public policy evaluation, the Korean government uses two approaches: a performance evalua- tion system introduced in 1962, and an audit and inspection system established in 1948. Performance evaluation has been carried out by organizations within or under the prime minister's office. Auditing and inspection are carried out by the Board of Audit, the supreme audit institution, and encompass auditing of public accounts and inspection of government agencies. The Board of Audit has grown and become stronger in recent years, and is now focusing on improvements in efficiency and transparency of audits and inspections. The Asian economic crisis of the late 1990s brought about new changes in evaluation practices in the executive branch. "The new government in Korea asserted that the national economic crisis, caused by for- eign exchange reserves, resulted from lack of efficiency of the public sector management. This assessment became an opportunity for reinventing government in Korea, which brought forth unprecedented restructur- ing of government organization as well as nongovernmental organization . . . " (Lee 2002, p. 194). With respect to public sector evaluation in Korea, there are now eight different approaches in place: · Institution evaluation, including evaluation of major policy measures, policy implementation capacity, and public satisfaction surveys of government services · Evaluation of major programs and projects, including a select number of key projects, chosen accord- ing to importance to the ministry, consistency with government policies, and importance to the public · Policy implementation capability evaluation, involving self-evaluation in the ministries, as well as an evaluation of an institution's ability to reform, innovate, and improve services · Survey of public satisfaction with major policy measures and administrative services, polling public satisfaction with major government policies, programs, and services · Special project evaluation, including, for example, state tasks and deregulation projects · Ministries' internal evaluation or self-evaluation, including evaluations of major target policy measures and programs, and government innovation efforts by each ministry · Evaluation of major policy measures and programs · Evaluation of government innovation efforts by every ministry. While Korea has made much progress in monitoring and evaluation, challenges remain. Cooperation and coordination between M&E institutions need strengthening. There has been excessive centralization of pol- icy analysis and evaluation as well as audit and inspection. Korea still lacks sufficient numbers of profes- sional and skilled personnel trained in M&E. Finally, more could be done to improve the effectiveness of post-evaluation proposals, which currently are not legally binding. Source: Lee 2002. 32 Ten Steps to a Results-Based Monitoring and Evaluation System political systems, strong empirical traditions, civil servants trained in the social sciences (as opposed to strict legal training), and efficient administrative systems and institutions. Indeed, building results- based M&E systems is primarily a political activity with some asso- ciated technical dimensions. Countries with high levels of expenditure on education, health, and social welfare also adopted evaluation mechanisms that then spilled over into other areas of public policy. Evaluation must satisfy a need. "What is involved is a complex mixture of institutional pre- conditions, political culture, exposure to intellectual traditions, as well as sectoral concerns dominating the political discussion . . . " (Furubo, Rist, and Sandahl 2002, p.16). Special M&E Challenges Facing Developing Countries The challenge of designing and building a results-based M&E system in a developing country is difficult and not to be underestimated. The construction of such a system is a serious undertaking, and will not happen overnight. However, it is also not to be dismissed as being too complicated, too demanding, or too sophisticated for a develop- ing country to undertake. All countries need good information sys- tems so they can monitor their own performance--developing coun- tries no less than others. Developing countries building their own results-based M&E sys- tems face challenges both similar to and different from those of de- veloped countries. Demand for and ownership of such a system--the most basic requirement--may be more difficult to establish in devel- oping countries. For example, a recent World Bank and African De- velopment Bank study found that " . . . the key constraint to success- ful monitoring and evaluation capacity development in Sub-Saharan Africa is lack of demand. Lack of demand is rooted in the absence of a strong evaluation culture, which stems from the absence of per- formance orientation in the public sector" (Schacter 2000, p. 15). With respect to demand, then, a minimum of interested stakeholders and commitment is necessary for such a system to be established and take hold in any country--whether developed or developing. In contrast to developed countries, developing countries may find it more challenging to do longer-term strategic economic, investment, and policy planning. Weak political will and institutional capacity may slow progress. Difficulties in interministerial cooperation and Introduction: Building a Results-Based Monitoring and Evaluation System 33 coordination can impede progress toward strategic planning, too. In- deed, lack of sufficient governmental cooperation and coordination can be a factor in both developed and developing countries. Highly placed champions who are willing to assume the political risks in advocating results-based M&E are also needed--again em- phasizing the political nature of building such systems. Sometimes they are present, as in the case of Egypt (Minister of Finance), Zam- bia (Secretary to the Cabinet), and the Kyrgyz Republic (Minister of Health), while in other instances, such as Bangladesh, they are lack- ing. The presence of a national champion can go a long way toward helping a country develop and sustain M&E systems. Many developing countries are still struggling to put together strong, effective institutions. Some may require civil service reform, or reform of legal and regulatory frameworks. They are being sup- ported by the international development community in improving many of these basic building blocks. Trying to build institutions, un- dertake administrative and civil service reforms, and revamp legal and regulatory codes--while at the same time establishing M&E sys- tems--can be quite a challenge. However, it should be remembered that instituting M&E systems can help better inform and guide the government in undertaking needed reforms in all of these areas. Developing countries must first have, or establish, a basic founda- tion--a traditional implementation-focused M&E system. Some de- veloping countries are moving in this direction. Establishing a foun- dation requires basic statistical systems and data, as well as key budgetary systems. Data and information must be of appropriate quality and quantity. Developing countries--like developed ones-- need to know their baseline conditions, that is, where they currently stand in relation to a given program or policy. Capacity in the workforce is needed to develop, support, and sus- tain these systems. Officials need to be trained in modern data collec- tion, monitoring methods, and analysis. This can be difficult for many developing countries. For example, there is a severe shortage of local capacity in Sub-Saharan African countries, compounded by the emigration of well-qualified people out of the region (Schacter 2000, p. 8). Technical assistance and training for capacity and institutional de- velopment may be required. Donors are often willing to finance and support such activities, and share lessons of best practice.2 At the 34 Ten Steps to a Results-Based Monitoring and Evaluation System same time, donors should try to harmonize their evaluation require- ments relative to recipient countries. As part of the donor effort to support local capacity in developing countries, donors are also moving to create development networks-- new computer on-line networks and participatory communities that share expertise and information. " . . . [I]t can still be argued that circumstances in Bangladesh, China, Costa Rica or Mali are unique and distinct, and that the experience of one country will not neces- sarily translate to another. But once it is accepted that there is very little generic development knowledge--that all knowledge has to be gathered and then analyzed, modified, disassembled and recombined to fit local needs--the source is immaterial. The new motto is: `Scan globally, reinvent locally'" (Fukuda-Parr, Lopes, and Malik 2002, p. 18). Developing countries will need to establish a political and adminis- trative culture characterized by accountability and transparency, con- cern for ethics, and avoidance of conflicts of interest. Reformers need to be aware, though, that any attempts to shed light on resource allo- cation and actual results through the adoption of an M&E system may meet with political resistance, hostility, and opposition. In addi- tion, given the nature of many developing country governments, building an M&E system could lead to considerable reshaping of political relationships. Creation of a more mature M&E system requires interdependency, alignment, and coordination across multiple governmental levels. This can be a challenge because, in many developing countries, gov- ernments are loosely interconnected, and are still working toward building strong administrative cultures and transparent financial sys- tems. As a result, some governments may have only vague informa- tion about the amount and allocation of available resources, and whether resources are, in fact, used for the purposes intended. Measuring government performance in such an environment is an approximate exercise. Developed and developing countries alike are still working toward linking performance to a public expenditure framework or strategy. If these linkages are not made, there is no way to determine if the budgetary allocations in support of programs are ultimately support- ing a success or a failure. Furthermore, there would be no means of providing feedback at interim stages to determine if fiscal adjust- Introduction: Building a Results-Based Monitoring and Evaluation System 35 ments could be made to alter projects or programs, and thereby in- crease the likelihood of achieving the desired results. Some developing countries are beginning to make progress in this area. For example, in the 1990s, Indonesia started to link evaluation to the annual budgetary allocation process. "Evaluation is seen as a tool to correct policy and public expenditure programs through more direct linkages to the National Development Plan and the resource allocation process" (Guerrero 1999, p. 5). In addition, some developing countries--Brazil, Chile, and Turkey--have made progress with respect to linking expenditures to output and outcome targets. The government of Brazil also issues sepa- rate governmental reports on outcome targets (OECD 2002b). Many developing countries still operate with two budget sys- tems--one for recurrent expenditures and another for capital invest- ment expenditures. Until recently, Egypt's Ministry of Finance over- saw the recurrent budget and the Ministry of Planning oversaw the capital budget. Consolidating these budgets within one ministry made it easier for the government to consider a results-based M&E system to ensure the country's goals and objectives will be met. Attempting to institute a whole-of-government approach toward M&E--as in Australia, Canada, and the United States--may be too ambitious for some developing countries. Given the particular diffi- culties of establishing M&E systems in developing countries, adopt- ing an enclave or partial approach, in which a few ministries or de- partments first pilot and adopt M&E systems, may be preferable. For example, in the Kyrgyz Republic, a 2002 readiness assessment rec- ommended that the Ministry of Health--where some evaluation ca- pacity already exists--be supported as a potential model for eventual government-wide implementation of a results-based M&E system (Kusek and Rist 2003). M&E Experience in Developing Countries Many developing countries have made progress toward instituting M&E. Keeping in mind the many challenges facing developing coun- tries, boxes i.ix and i.x consider two examples: Malaysia and Uganda. Both countries have introduced new--albeit different-- measures to the budgetary process to make it more transparent, ac- countable, and results focused. The challenges facing developing countries are many. The coun- 36 Ten Steps to a Results-Based Monitoring and Evaluation System tries' approaches may differ and it may require a considerable period of time to arrive at a results-based M&E approach. But experience around the world shows that the foundation for evaluation is being built in many developing countries. (See annexes 4 and 5 for more on developing country efforts with respect to M&E.) Box i.ix Malaysia: Outcome-Based Budgeting, Nation Building, and Global Competitiveness Among developing countries, Malaysia has been at the forefront of public administration reforms, espe- cially in the area of budget and finance. These reforms were initiated in the 1960s as part of an effort by the government to strategically develop the country. The public sector was seen as the main vehicle of develop- ment, consequently the need to strengthen the civil service through administrative reform was emphasized. Budgetary reform focused on greater accountability and financial discipline among the various govern- ment agencies entrusted to carry out the socioeconomic development plans for the country. In addition to greater public sector accountability and improved budgetary system performance, the government under- took a number of additional reforms including improved financial compliance, quality management, pro- ductivity, efficiency in governmental operations, and management of national development efforts. Most recently, Malaysia's budget reform efforts have been closely linked with the efforts at nation build- ing and global competitiveness associated with Vision 2020--a program aimed at making Malaysia a fully developed country by the year 2020. With respect to budgetary reform, Malaysia adopted the Program Performance Budgeting System (PPBS) in 1969 and continued to utilize it until the 1990s. The PPBS replaced line-item budgeting with an outcome based budgeting system. While agencies used the program-activity structure, in practice implementation still resembled the line item budgeting and an incremental approach. In 1990, the government introduced the Modified Budgeting System (MBS) to replace the PPBS. Greater emphasis was placed on outputs and impact of programs and activities in government. Under PPBS, there were minimal links between outputs and inputs. Policies continued to be funded even when no results were being systematically measured. The MBS approach was further modified in 2001, when the country embarked on another complemen- tary reform by adopting a two-year budgeting system. The effect of this system will be known in several years time. Although Malaysia has been at the forefront of public administration and budget reforms, these reform efforts have not been smooth or consistent over the years. Nonetheless, the MBS was a bold initiative on the part of the Malaysian government, demonstrating foresight, innovativeness, dynamism, and commit- ment to ensure value for money in the projects and policies being implemented. Source: World Bank 2001b. Introduction: Building a Results-Based Monitoring and Evaluation System 37 With the growing global movement to demonstrate accountability and tangible results, many more developing countries can be expected to adopt results-based M&E systems in the future. The international donor community's focus on development impact Box i.x Uganda and Poverty Reduction--Impetus toward M&E "The government of Uganda has committed itself to effective public service delivery in support of its poverty-reduction priorities. The recognition of service delivery effectiveness as an imperative of national development management is strong evidence of commitment to results, which is also evident in several of the public management priorities and activities that are currently ongoing" (Hauge 2001, p. 16). Over the past decade, Uganda has undergone comprehensive economic reform and has achieved macro- economic stabilization. Uganda developed a Poverty Eradication Action Plan (PEAP) in response to the Comprehensive Development Framework, and it is now incorporated into the Poverty Reduction Strategy Paper. The PEAP calls for a reduction in the absolute poverty rate from 44 percent (as of the late 1990s) to 10 percent by the year 2017. Uganda was the first country to be declared eligible and to benefit from Highly Indebted Poor Country (HIPC) measures. Most recently, Uganda qualified for enhanced HIPC relief in recognition of the effective- ness of its poverty reduction strategy, consultative process involving civil society, and the government's con- tinuing commitment to macroeconomic stability. Uganda has introduced new measures to make the budget process more open and transparent to internal and external stakeholders. The government is modernizing its fiscal systems, and embarking on a decentral- ization program of planning, resource management, and service delivery to localities. The Ministry of Finance, Economic Planning and Development (MFPED) is also introducing output-oriented budgeting. In addition, government institutions will be strengthened and made more accountable to the public. The country is still experiencing a number of coordination and harmonization difficulties with respect to M&E and the PEAP. "The most obvious characteristic of the PEAP M&E regime is the separation of poverty monitoring and resource monitoring, albeit both coordinated by the MFPED. The two strands of M&E have separate actors, reports and use different criteria of assessment. Financial resource monitor- ing is associated with inputs, activities and, increasingly, outputs, whereas poverty monitoring is based on analyzing overall poverty outcomes" (Hauge 2001, p. 6). Other M&E coordination issues revolve around the creation of a new National Planning Authority, and among the sector working groups. Regarding future challenges and M&E, Uganda faces the task of keeping track of and learning from its progress toward poverty reduction via the PEAP/National Poverty Reduction Strategy. M&E cannot be isolated from the decisionmaking practices and incentives that underpin national development systems and processes. Sources: Hauge 2001; World Bank 2002b. 38 Ten Steps to a Results-Based Monitoring and Evaluation System means that more donors will need to step in to ensure the necessary Developing countries de- assistance for developing countries to implement such systems. serve good governance no Instituting results-based M&E systems has been challenging for less than other countries. developed as well as developing countries--though developing coun- tries face special difficulties. There is no one correct path or ap- proach. Getting there takes commitment, effort, time, and resources. At the same time, one should continue to bear in mind that there are also costs to not instituting such systems and not responding to inter- nal and external stakeholder calls for accountability, transparency, and results. Chapter 1 Step 1: Conducting a Readiness Assessment Figure 1.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 11 2 3 4 5 6 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization In the introduction we examined new challenges in public sector management--calls for increased public accountability, better gov- ernance, and demonstrable results. We introduced a new public man- agement tool, the results-based monitoring and evaluation system, that can help policymakers respond to the increasing demands by NGOs; civil society; and national, multilateral, and international stakeholders for better performance. Finally, we reviewed the moni- toring and evaluation experience in developed and developing coun- tries, as well as the special challenges facing the developing world in building results-based M&E systems. In this chapter, we turn to Step 1 of our 10-step model (figure 1.1)--the readiness assessment. This step is a unique addition to the many M&E models that currently exist because it provides an ana- lytical framework to assess a given country's organizational capacity and political willingness to monitor and evaluate its goals, and de- 39 40 Ten Steps to a Results-Based Monitoring and Evaluation System velop a performance-based framework. This is a key step--unfortu- nately often missed or omitted--in helping developing countries, in particular, build their own results-based M&E systems. Specifically, this chapter addresses: (a) the importance of conduct- ing a readiness assessment; (b) the three main parts of the readiness assessment; (c) the eight key diagnostic areas that must be considered in any readiness assessment; (d) some examples of recent readiness assessments done in developing countries; and (e) lessons learned from these experiences. Annex I details a version of the readiness as- sessment step: "Assessing Results-Based Monitoring and Evaluation Capacity: An Assessment Survey for Countries, Development Institu- tions, and Their Partners" that countries can use for their own self- assessments. PART 1 Why Do a Readiness Assessment? Experts have devised a number of different models for building M&E systems, but often miss the complexities and nuances of the wider country context. The needs of the recipient country are often only vaguely understood by those experts trying to provide technical assistance. For all the good intentions to advance the design, cre- ation, and use of results-based M&E systems, too little emphasis is placed on existing political, organizational, and cultural factors and contexts. Most of the existing models start by jumping straight into building It will be a constant theme a results-based M&E system--without even knowing where a given of this handbook that country stands in relation to a number of critical factors, including building a results-based organizational roles, responsibilities, and capabilities; incentives and M&E system is first and demands for such a system; ability of an organization to sustain sys- foremost a political ac- tems; and so forth. There are a few models that pose key readiness tivity with technical di- questions. (See Mackay 1999 and World Bank 2003a.) mensions rather than vice Most experts look at the "what" questions--what are the goals? versa. what are the indicators?--and not the "why" questions: Why do we want to measure something? Why is there a need in a particular country to think about these issues? Why do we want to embark on building sustainable results-based M&E systems? To answer these "why" questions, there is a considerable amount of preparatory work to do before the actual construction of a results- based M&E system. That preparatory work takes the form of the Conducting a Readiness Assessment 41 readiness assessment presented here. We will walk through, step-by- A readiness assessment is step, some of the important issues, concerns, and questions that like constructing the foun- should be addressed before embarking on building an M&E system. dation for a building. A Some might also pose the question: How does a readiness assess- good foundation provides ment differ from a needs assessment? Are they not the same thing? In support for all that is above fact, they are not. A needs assessment assumes that there is some fun- it. It is below ground, not damental, underlying question as to whether governments need such seen, but critical. systems. A readiness assessment assumes that governments need to have these systems, and addresses whether governments are actually ready and able to move forward in building, using, and sustaining the systems. For example, what is the government's capability with respect to M&E in general? Does it simply measure outputs, or is the government in a position to move beyond measuring outputs to measuring outcomes? (It should also be remembered that studying organizational capacity is not enough. These are just a few of the key questions and concerns that only a readiness assessment can address and answer.) A readiness assessment provides the analytical frame- work for rating a country's ability to monitor and evaluate its progress in achieving designated development goals. It does this by assessing a country's current understanding, capacity, and use of ex- isting monitoring and evaluation systems. Three Main Parts of the Readiness Assessment The readiness assessment is a diagnostic aid that will help determine where a given country3 stands in relation to the requirements for establishing a results-based M&E system. It is composed of three main parts. Incentives and Demands for Designing and Building a Results- Based M&E System It is important to determine whether incentives exist--political, institutional, or personal--before beginning to design and build a results-based M&E system. There are five key questions related to incentives: · What is driving the need for building an M&E system--legisla- tive or legal requirements, citizen demand, donor requirements (National Development Plan, National Poverty Reduction Strategy, or others), or political or public sector reform? · Who are the champions for building and using an M&E sys- tem--government, parliament, civil society, donors, others? · What is motivating those who champion building an M&E sys- 42 Ten Steps to a Results-Based Monitoring and Evaluation System tem--a political reform agenda, pressures from donors, a per- sonal political agenda, or political directive? · Who will benefit from the system--politicians, administrators, civil society, donors, citizens? · Who will not benefit from building an M&E system--politi- cians, administrators, civil society, donors, citizens? Are there counterreformers inside or outside the political system? Roles and Responsibilities and Existing Structures for Assessing Performance of the Government A readiness assessment will enable one to gauge the roles and responsibilities and existing structures available to monitor and evaluate development goals. · What are the roles of central and line ministries in assessing per- formance? · What is the role of parliament? · What is the role of the supreme audit agency? · Do ministries and agencies share information with one another? · Is there a political agenda behind the data produced? · What is the role of civil society? · Who in the country produces data? - At the national government level, including central ministries, line ministries, specialized units or offices, including the national audit office - At the subnational or regional government level, including provincial central and line ministries, local government, NGOs, donors, and others · Where in the government are data used? - Budget preparation - Resource allocation - Program policymaking - Legislation and accountability to parliament - Planning - Fiscal management - Evaluation and oversight. Capacity Building Requirements for a Results-Based M&E System The readiness assessment also includes a review of a country's cur- rent capacity to monitor and evaluate along the following dimen- sions: technical skills; managerial skills; existence and quality of data systems; available technology; available fiscal resources; and institu- Conducting a Readiness Assessment 43 tional experience. This is an important part of the assessment in de- veloping countries, because it can help identify any gaps in capacity needed to build and sustain results-based M&E systems. Such an assessment also directs one to examine existing or possible barriers to building an M&E system, including a lack of fiscal re- sources, political will, political champion, expertise, strategy, or prior experience. A number of key questions need to be considered: · What are the skills of civil servants in the national government in each of the following five areas: - Project and program management - Data analysis - Project and program goal establishment - Budget management - Performance auditing? · Is there any technical assistance, capacity building, or training in M&E now underway or that was done in the past two years for any level of government (national, regional, or local)? Who pro- vided this help and under what framework or reform process? · Are there any institutes, research centers, private organizations, or universities in the country that have some capacity to provide technical assistance and training for civil servants and others in performance-based M&E? Now we will build on this material and explore the eight key areas covered by a readiness assessment in more detail. PART 2 The Readiness Assessment: Eight Key Questions The readiness assessment is a diagnostic tool that can be used to de- termine whether the prerequisites are in place for building a results- based M&E system. It is intended to assist and benefit individual governments, the donor community, and their many development partners involved in public sector reform.4 The readiness assessment provides a guide through the eight areas that must be considered and explored in determining a given coun- try's or organization's ability and willingness to adopt and move for- ward with a results-based M&E system. 44 Ten Steps to a Results-Based Monitoring and Evaluation System What Potential Pressures Are Encouraging the Need for the M&E System within the Public Sector and Why? It is important to know where the demand for creating an M&E sys- tem is emanating from and why. Are the demands and pressures com- ing from internal, multilateral, or international stakeholders, or some combination of these? These requests will need to be acknowledged and addressed if the response is to be appropriate to the demand. As noted in the introduction, internal demands may arise from calls for reforms in public sector governance and for better account- ability and transparency. Anti-corruption campaigns may be a moti- vating force. Or political opponents may not trust the government's intentions or actions. Externally, pressures may arise from the donor community for tan- gible development results for their investments. International organiza- tions, such as the European Union, expect a feedback system on pub- lic sector performance via M&E for each of the accession countries. The competitive pressures of globalization may come into play, and the rule of law, a strong governance system, and clearly articulated rules of the game are now necessary to attract foreign investment. Fi- nancial capital and the private sector are looking for a stable, transpar- ent investment climate, and protection of their property and patents, before committing to invest in a country. There are a multitude of pressures that governments may need to respond to, and these will drive the incentives for building a results-based M&E system. Who Is the Advocate for an M&E System? Champions in government are critical to the sustainability and suc- cess of a results-based M&E system. A highly placed government champion can be a strong advocate for more well-informed decision- making, and can help diffuse and isolate attacks from counterreform- ers who will have vested interests in averting the construction of such a system. Within a given organization, there are individuals or groups who will likely welcome and champion such an initiative, while others may oppose or even actively counter the initiative. It is important to know who the champions are and where they are located in a gov- ernment. Their support and advocacy will be crucial to the potential success and sustainability of the M&E system. However, if the emerging champion is located away from the cen- ter of policymaking and has little influence with key decisionmakers, Conducting a Readiness Assessment 45 it will be difficult, although not impossible, to envision an M&E sys- tem being used and trusted. It will be hard to ensure the viability of the system under these circumstances. Viability is dependent upon the information being viewed as relevant, trustworthy, useable, and timely. M&E systems with marginally placed champions who are pe- ripheral to the decisionmaking process will have a more difficult time meeting these viability requirements. What Is Motivating the Champion to Support Such an Effort? Constructing a results-based M&E system is an inherently political Understanding political act entailing both political risks and benefits. On the risk side, pro- motivation is critical to un- ducing information on government performance and strengthening derstanding how an M&E accountability are not politically neutral activities. On the benefit system will be perceived by side, champions may find rewards and recognition at the institutional stakeholders, how and why and individual levels. Champions may be motivated by a sense of certain persons or organ- public responsibility. Champions may also find favor with parlia- izations will take the politi- ments, public and private stakeholders, civil society, and the interna- cal risk while others will tional donor community by delivering on promises, being perceived not, and what those cham- as a reformer (a source of political capital), and demonstrating ac- pioning such a system will countability and results. need to defend the initiative Who Will Own the System? Who Will Benefit from the System? and succeed. How Much Information Do They Really Want? Politics is not the only factor often overlooked in building M&E sys- tems. Frequently, a careful institutional assessment is not made--in particular, one that would reflect the real capacity of the users to ac- tually create, utilize, and sustain the system. A carefully done readiness assessment helps provide a good under- standing of how to design the system to be responsive to the informa- tion needs of its users, determine the resources available to build and sustain the system, and assess the capacities of those who will both produce and use the information. Understanding these issues helps to tailor the system to the right level of complexity and completeness. For a results-based M&E system to be effectively used, it should provide accessible, understandable, relevant, and timely information and data. These criteria drive the need for a careful readiness assess- ment prior to designing the system, particularly with reference to such factors as ownership of the system, and benefits and utility to key stakeholders. From a technical perspective, issues to be addressed include the capacity of the government or organization to collect and 46 Ten Steps to a Results-Based Monitoring and Evaluation System analyze data, produce reports, manage and maintain the M&E sys- tem, and use the information produced. Thus, the readiness assessment will provide important information and baseline data against which capacity-building activities--if neces- sary--can be designed and implemented. Furthermore, there is an absolute requirement to collect no more information than is required. Time and again, M&E systems are de- signed and are immediately overtaxed by too much data collected too often--without sufficient thought and foresight into how and whether such data will actually be used. Complexity and overdesign are constant concerns. There will also be a continuous erosion in the system that will need to be addressed. And stakeholders may try to pull the system in too many different directions at once. In short, little in the political arena remains the same. Keeping the M&E sys- tem up and running will demand vigilance and care (yet another rea- son why champions are necessary). How Will the System Directly Support Better Resource Allocation and the Achievement of Program Goals? Monitoring and evaluation is not an end unto itself. It is a tool to be used to promote good governance, modern management practices, innovation and reforms, and better accountability. When used prop- erly, these systems can produce information that is trustworthy, trans- parent, and relevant. M&E systems can help policymakers track and improve the outcomes and impacts of resource allocations. Most of all, they help governments and organizations make more well-informed decisions and policies by providing continuous feedback on results. Experience shows that the creation of a results-based M&E system often works best when linked with other public sector reform pro- grams and initiatives, such as creating a medium-term public expen- diture framework, restructuring public administration, or construct- ing a National Poverty Reduction Strategy. Linking the creation of M&E systems to such initiatives creates interdependencies and rein- forcements that are crucial to the overall sustainability of the systems. The readiness assessment can provide a road map for determining whether such links are structurally and politically possible. How Will the Organization, the Champions, and the Staff React to Negative Information Generated by the M&E System? It is difficult to have a functioning M&E system in an organizational or political climate characterized by fear. M&E systems will in- Conducting a Readiness Assessment 47 evitably (even if infrequently) produce data that may be embarrass- ing, politically sensitive, or detrimental to those in power. In a similar way, the information can also be detrimental to units and individuals in an organization. ("Punishing the messenger" is not an unknown occurrence in organizations.) If it is clear from the readiness assessment that only politically popular or "correct" information will be allowed to emanate from the M&E system, the system is vulnerable and compromised from the beginning. It will not be seen as credible by those outside the or- ganization. It will come to be seen as a hollow exercise. In such a po- litical setting, it is important to build the system carefully and slowly. Finding units that will risk potentially detrimental information--in- cluding unfavorable information about their own performance--is perhaps the best that can be achieved. If such units are not present, there is little rationale or justification for proceeding further to de- sign such a system. An emphasis on traditional implementation mon- itoring will have to suffice. Governments willing to use performance information to make pol- A readiness assessment icy generally have achieved some level of democracy and openness. will help identify the bar- But even in these countries, there is often a reluctance to measure and riers and obstacles--struc- monitor because of fears that the process will bring bad news to lead- tural, cultural, political, or ership and stakeholders alike. There are real political limitations to individual--in a given be recognized in building such systems. organization. Not all barriers can be addressed simultaneously in the design of the system. However, not recognizing the presence of these barriers and addressing them as soon as possible creates the risk of a level of resistance greater and longer than may have been necessary. It is a strategic decision as to how much time and energy should be spent on removing barriers as opposed to using that same finite time and energy to strengthen champions and support emerging opportunities. We strongly lean toward the latter. Where Does Capacity Exist to Support a Results-Based M&E System? Performance data and information can be found in many places. The readiness assessment provides a useful guide to determining where such information and data can be found. For instance, are there any organizational units within the government that already have moni- toring and evaluation capacity and that can undertake evaluations? What data systems can be found within, or are available to, the cen- tral and sector or line ministries of the government responsible for 48 Ten Steps to a Results-Based Monitoring and Evaluation System planning? This can include budget data, output data, outcome or im- pact data, performance audits, financial audits, project and program completion reports, and donor data information. Outside the govern- ment, NGOs, universities, research institutes, and training centers may also provide part of the necessary technical capacity to support a results-based M&E system. How Will the M&E System Link Project, Program, Sector, and National Goals? One of the main functions of the readiness assessment is to determine the opportunities for and risks of linking information across the gov- ernment in an aligned fashion. In an ideal situation, project level per- formance data would be fed into and linked to program assessments that, in turn, would be linked to sectoral, regional, and national goals and targets. In other words, staff at each level would have a clear "line of sight" into, or understanding about, each of the other levels and how they relate to one another. Results-based M&E at the project level that is not clearly aligned with program goals is not useful beyond the restricted information for a given project. Information must flow freely between levels to be truly useful. Each level must help inform the next level to achieve the desired results. It is important, as well, to ensure that within a level, there is a commitment to horizontally use and share information from the collection and analysis of data. The goal is to create an M&E sys- tem that is transparent and aligned from one level to the next. Infor- mation should flow up and down in a governmental system, rather than being collected, stored, and used at one level--but never shared across levels. A free flow of information can help ensure that policies, programs, and projects are linked and coordinated. Ultimately, the real question is whether the system can address the need at every level to be both producers and consumers of results-based information. PART 3 Readiness Assessments in Developing Countries: Bangladesh, Egypt, and Romania The readiness assessment can help governments, donors, and their partners address the challenges of the training, organizational capac- ity building, and sequencing of efforts that will be needed to design and construct results-based M&E systems. It provides the basis for an action plan to move forward in the country. Conducting a Readiness Assessment 49 A readiness assessment should begin with a look at the data that are currently reported by traditional implementation-focused M&E systems, and whether public expenditure, financial, data, or procure- ment reviews have been done. Is the country moving toward eco- nomic, legal, and political reform; greater democracy and openness; more accountability and transparency? (Occasionally, one finds that several different diagnostic surveys are being undertaken simultane- ously. In the Kyrgyz Republic in early 2002, for example, there was a Country Performance Portfolio Review, a Public Expenditure Re- view, and a Monitoring and Evaluation Review going on at the same time.) After reviewing where the country stands with regard to public management reforms, a country or field mission should then be undertaken. While in the country, information is gathered in the field from key informants, including government officials, members of civil society, and NGOs. It is important to talk with ministers and a broad range of sector-level officials. One never knows where one will find a champion who is interested in having a performance-based data sys- tem that will enhance policymaking. Ideally, the readiness assessment should be undertaken by someone familiar with M&E capacity building. Readiness Assessments: Three Developing Country Cases Let us look now at three actual examples from the developing world--Bangladesh, Egypt, and Romania--to see how the readiness assessment can inform and shape efforts to build results-based M&E systems (boxes 1.1 through 1.3). We will also draw lessons from these experiences that may be applicable to other developing countries. PART 4 Lessons Learned What are the lessons that can be drawn from these three readiness as- sessment examples from the developing world? Incentives and Demands for Designing and Building a Results- Based M&E System It is most important to understand the situation in a given country in the eight areas outlined in the readiness assessment. Had the assessment not been conducted in Bangladesh, for example, efforts to design and build a results-based M&E system might have moved 50 Ten Steps to a Results-Based Monitoring and Evaluation System Box 1.1 The Case of Bangladesh--Building from the Bottom Up In the course of implementing the readiness assessment, Bangladesh posed a considerable challenge with re- spect to its readiness to design and build a results-based M&E system. In 2001, Bangladesh was ranked the most corrupt country of the 91 countries monitored by Transparency International, with the most corrupt public sector listed as the law enforcement agencies, followed by education, local government, and health. In 2002, Bangladesh was again listed as the most corrupt of the 102 countries monitored. Corrupt systems keep information out of the public domain--and this is a major obstacle to M&E. The readiness assessment found no champion for M&E anywhere in the national government, including central and sector ministries. No reform initiatives could be identified that could create incentives for linking these reforms to the creation of an M&E system. Furthermore, there were no legal or regulatory requirements for the use of M&E that could be identified. There were some monitoring systems in rural parts of the country for education, electrification, and food subsidies. There was also some evidence that NGOs and the donor community were actively monitor- ing for results of development projects, but this had not influenced the government to do the same. The Bangladesh Bureau of Statistics was found to be a strong state agency. If and when the government moves toward developing a results-based M&E system, the bureau could play a central role in the collection and analysis of data. In terms of technical capability, the readiness assessment found weak capacity for M&E, and minimal technical training capacity in universities and research centers. The assessment also indicated minimal organ- izational experience in the national government with respect to managing credible information systems. As a result of the readiness assessment, we found that it was not realistic and feasible to introduce a re- sults-based M&E system into the national government at that time. Strong political support and sustained institution capacity building will be needed before such an initiative can be undertaken. There is hope on the horizon for Bangladesh. Subsequent to the readiness assessment, the government developed a National Poverty Reduction Strategy that will include M&E components. The readiness assess- ment recommended five strategies to donors and NGOs working in Bangladesh to strengthen some of their capacity and work in small, targeted ways. Source: World Bank 2002c. Conducting a Readiness Assessment 51 Box 1.2 The Case of Egypt--Slow, Systematic Moves toward M&E One of the most important components of assessing a country's readiness to introduce results-based M&E is whether a champion can be found who is willing to take on ownership of the system. Conducting the readi- ness assessment uncovered significant interest in Egypt on the part of many senior government officials for moving toward a climate of assessing performance. The president himself has called for better information to support economic decisionmaking. The Minister of Finance was found to be a key champion for the government of Egypt's move to a results focus. This minister was well versed in the international experience of other countries, such as Malaysia and OECD member countries. The minister underscored the importance of giving increased attention to improv- ing the management of public expenditures by moving forward with a set of pilots to demonstrate how re- sults-based M&E could be used to better manage budgetary allocations. The Minister of Finance will play a key leadership role in any effort to introduce results-based M&E in Egypt. A number of other senior officials were identified who could play important roles. The First Lady of Egypt, who chairs the National Council for Women, is developing a system to monitor and evaluate efforts across many ministries to enhance the status and condition of women. However, for an M&E effort to be successful and sustainable, there must be a "buy-in" (or a sense of ownership) from line ministers who are responsible for resource expenditures and overseeing the implementation of specific programs. The team found interest in monitoring and evaluation for results on the part of several line ministers, including the Minister of Electricity and Energy, and the Minister of Health. The readiness assessment also revealed a high level of capacity in Egypt to support the move toward a results-based strategy. A number of individuals with evaluation training were identified at the University of Cairo, the American University of Cairo, and private research organizations. In addition, the Central Agency for Public Mobilization and Statistics, and the Cabinet Information Decision Support Center have key roles in collecting, analyzing, and disseminating data to be used by both government and nongovernment re- searchers and policymakers. A key criterion for a successful shift toward results is the development of a well-communicated and exe- cutable strategy. The diagnostic identified a fragmented strategy for moving the effort forward. A set of pilots had tentatively been identified, yet there were few, if any, criteria for establishing these as performance pilots. Nor was there a management structure set up within the government to effectively manage the overall effort. The Minister of Finance, however, had begun to define an approach that, if implemented, would pro- vide the necessary leadership to move the effort forward. The minister was definite in his desire to move slowly and to nurture the pilots, learning along the way. The results of this readiness assessment suggest that the government of Egypt is prepared to take ownership of the effort and to systematically and slowly begin to introduce the concepts of results management. Visible ca- pacity exists that can be drawn upon to sustain the effort. Significantly, there is obvious political support to provide the necessary leadership. (The complete Egypt Readiness Assessment can be found in annex II.) Source: World Bank 2001c. 52 Ten Steps to a Results-Based Monitoring and Evaluation System Box 1.3 The Case of Romania--Some Opportunities to Move toward M&E Romania is in negotiations with the European Union to gain accession, and hopes to become a member by 2007. The government has a clear political commitment to reform, and has developed a medium-term eco- nomic strategy. Romania also has a work force skilled in data collection and use. In this sense, it is ahead of many other developing and transition economies. At the same time, though, Romania continues to suffer from the communist legacy in a number of ways, such as a continued central planning mentality in some parts of the government, weak governmental insti- tutions, few government officials trained in modern public management principles and practices, and an inexperienced civil society with no tradition of actively participating in the business of government. The readiness assessment revealed other barriers to moving toward M&E. These included a lack of understanding within Romania's public sector as to what is entailed in the development of a performance- oriented management culture, and conflicts with other overall government priorities. Romania is conducting a set of budget performance pilots in which 5 agencies have been asked to submit a set of performance measures as an annex to the annual budget. The pilot program began with 5 governmental agencies, moving to 8 in the following year, 13 in the year thereafter, and finally to all agencies. At the time of the readiness assessment, the government was still in the pilot phase with a new budget that included 13 pilots. The pilots that focused on allocating funds to agencies based on performance indicators were largely ignored by government managers and not taken seriously by the parliament. However, the pilots did repre- sent a focal point for learning how to develop appropriate performance indicators to monitor the effective- ness of the annual budget. The Minister of Finance appears to be a strong champion of the effort, and could provide the necessary political leadership to initiate and sustain a larger results-based management effort. Two additional potential champions were identified, including the Minister of Justice and a counselor to the prime minister. Both are leading efforts to improve management in the Romanian government, and they recognize the importance of reporting on the success of strategies. The government's commitment to move toward M&E is supported by a framework of new laws, the move toward EU accession, and rising expectations on the part of civil society. For example, one change in the legal framework includes a set of laws assisting a drive toward e-administration. This initiative could be a potential vehicle for improving government transparency and providing civil society with results of the government's reform program. Developing e-administration can be a potent instrument for government accountability. Finally, the readiness assessment suggests a number of opportunities to support the introduction of re- sults-based M&E. The ongoing performance budgeting effort and other government reforms could provide a significant focus on, and catalyst for, results. There is also sufficient high level political leadership to jump-start M&E within at least three pilot areas: budget, anticorruption, and poverty reform. Source: World Bank 2001d. Conducting a Readiness Assessment 53 forward in an environment that had few of the necessary precon- ditions for M&E. Similarly, in both Egypt and Romania, the readi- ness assessment provided vital information regarding likely entry points for designing and building a results-based M&E system that had the benefit of strong champions and a larger reform environment. There must be an acknowledged and publicized mandate for mov- ing toward a results-oriented climate prior to introducing programs. As noted earlier, this can come about as a result of internal or exter- nal initiatives and forces. For example, the mandate might include a budget management reform law, EU accession, pressure from a con- cerned citizenry, the need to reduce burdensome civil service payrolls, or a desire to make good on political promises. A sustained source of demand for performance information should be encouraged and supported, putting the government on notice that it will need to demonstrate results--that is, governments will need to demonstrate that the policies and programs being implemented are meeting expectations. Governments need prodding to ensure that re- porting results becomes a regular and routine activity. A successful results-based M&E system must have sustained lead- ership. While it is important to have good program managers over- seeing the implementation of government programs and projects, there must also be strong political support at the very highest levels of government. The country, through its government, must be in the driver's seat in developing these systems to ensure ownership. With- out a strong, well placed champion who is willing to take on the ownership of a results-based M&E system, the system will not be built or used. Roles and Responsibilities and Existing Structures for Assessing Government Performance High turnover among government officials represents a challenge to building M&E systems. Frequent personnel changes in ministries make it difficult to identify and keep working with champions. This might be another reason to look for additional champions in civil so- ciety, NGOs, or in parliament. In many developing countries, different ministries and parts of government are going to be at different stages in their ability to mon- itor and evaluate. One should not necessarily assume that the whole government will be moving in tandem. There inevitably will be some 54 Ten Steps to a Results-Based Monitoring and Evaluation System sequencing and staggering with respect to building M&E systems. The readiness assessment can serve as a guide through the political system, and help identify the ability level of government ministries and agencies to monitor and evaluate. One should focus on nurturing those parts of the government that are in a position to move faster toward developing an M&E culture. Clear links between the budget and other resource allocation deci- sions are also necessary in making the shift to a results-based culture. In most governments, there is more than one agency working on a particular program. The readiness assessment can help identify over- laps among agencies so that overall program performance can be more effectively and efficiently measured and achieved. In effect, the readiness assessment can be a guide toward brokering differences be- tween agencies doing the same or similar tasks. Government policymakers need to be in communication and work in partnership with those responsible for information gathering and dissemination--particularly in areas such as the MDGs. Separate uni- verses of political action, support, and capacity building will not work. The M&E system needs to be integrated into the policy arena of the MDGs so that it will be clear to all stakeholders why it is im- portant to collect data, how the information will be used to inform the efforts of the government and civil society to achieve the MDGs, and what information needs to be collected. Capacity Building Requirements for a Results-Based M&E System Policy and management decisions should be based on reliable infor- mation. Bangladesh, Egypt, and Romania--like so many developing countries--lack sufficient capacity and many of the necessary re- sources for building M&E systems. This is not an insurmountable obstacle. Expertise, strategy, and experience can be acquired with time and money. However, lack of political will and champions will impede any move toward an M&E culture. The country must eventually have its own capacity to design, im- plement, and use a results-based M&E system. It is not enough to ac- quire skills such as social research, public management, statistics, or data management via consulting contracts from the international community. These skills must, in some way, come to reside within the country--and be available for contributing to a program of regularly assessing the performance of government. If these skills are not pres- Conducting a Readiness Assessment 55 ent in sufficient quantities, a concerted capacity-building program is necessary. Countries will need to build the capacity to implement pockets of innovation that can serve as beginning practices or pilot programs. The ability to test and pilot will become particularly important when we examine the selection of key performance indicators in chapter 3. One of the challenges in designing and building M&E systems is that there are so many different donors often asking the government to report on the same development goal. The readiness assessment can be used as a tool for donor coordination of M&E systems, and attendant capacity- and institution-building activities. Such coordina- tion can help the country make the best use of donor resources, in particular by avoiding the pitfalls of duplication, underfunding, or mismatch of priorities. The challenges of designing and building a results-based M&E system in a developing country are not to be underestimated. The construction of such a system is a serious undertaking and will not happen overnight. However, it is also not to be dismissed as too complicated, demanding, or sophisticated for a developing country to initiate. All countries need good information systems so they can monitor their own performance--developing countries no less than others. Consequently, assisting developing countries in achieving this capacity merits the time and attention of country officials and their development partners. Chapter 2 Step 2: Agreeing on Outcomes to Monitor and Evaluate Figure 2.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 22 3 4 5 6 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization Setting goals is part of the governmental decisionmaking process at "If you do not know where every level. All governments have goals--although not all have M&E you are going, any road capacity. Assuming that a country or organization is in fact in a posi- will take you there." tion to move forward in building a results-based M&E system, the (Alice's Adventures in Wonderland, next step is to choose and agree on the outcomes (derived from the Lewis Carroll, 1865) goals) to monitor and evaluate (figure 2.1). Knowing where you are going before you get moving is key. Specifically, this chapter addresses (a) the importance of outcomes; (b) issues to consider in choosing outcomes to monitor and evaluate; (c) the importance of building a participatory and consultative process involving main stakeholders; and (d) the overall process of setting and agreeing on outcomes. Examples for consideration and discussion are also included. The Importance of Outcomes At the outset, it is important to distinguish between goals and out- comes. Goals are generally long term, such as the MDGs that were 56 Agreeing on Outcomes to Monitor and Evaluate 57 reviewed earlier. From goals we move to outcomes, which, in the Outcomes are usually not MDG example, are of intermediate time frame (five to ten years). directly measured, only From outcomes we derive targets that are generally short-range--in reported on. the MDG context, about one to three years. Why is it important to emphasize outcomes at this stage? Why not move directly to setting indicators? Because establishing outcomes will illustrate what success looks like. By contrast, indicators are only relevant when they measure against an objective. Thus, measuring in- dicators will show the progress made toward reaching the intended objectives. Decisionmakers and stakeholders are positioned to make the in- tended outcomes of governmental action as explicit as possible. One cannot set indicators before determining outcomes because it is the outcomes--not the indicators--that will ultimately produce the bene- fits. Outcomes will demonstrate whether success has been achieved. In short, outcomes will show which road to take. Setting outcomes is essential in building a results-based M&E system. Building the system is basically a deductive process in which inputs, activities, and outputs are all derived and flow from the setting of outcomes. Indicators, baselines, and targets (covered in subsequent chapters), all crucial elements of the perform- ance framework, are derived from and based on the setting of out- comes. Issues to Consider in Choosing Outcomes to Monitor and Evaluate What are the strategic priorities? What are the desired outcomes? These are the questions that every organization, every level of gov- ernment, and the interested parties in civil society can be asking--of themselves and others. We focus in the following primarily on how this relates to the national government. Every country has finite budgetary resources and must set priori- ties. Consequently, it is important to keep the following distinction in mind: One budgets to outputs and manages to outcomes. There are many issues to consider in choosing outcomes to moni- tor and evaluate. For example, outcomes could be linked to interna- tional economic development and lending issues, including a Na- tional Poverty Reduction Strategy, a National Development Plan, the HIPC Initiative, or the MDGs. If there is an EU accession plan for the country, decisionmakers need to examine a host of socioeconomic and political benchmarks, 58 Ten Steps to a Results-Based Monitoring and Evaluation System and articulate specific desired outcomes to meet them, to formally join this important regional bloc. At the country level, there could already be some stated national, re- gional, or sectoral goals. Also, political and electoral promises may have already been made that specify improved governmental performance in a given area. In addition, there may be citizen polling data indicating particular societal concerns. Parliamentary actions and authorizing legislation are other areas that should be examined in determining desired national goals. There may also be a set of simple goals for a given project or program, or for a particular region of a country. From these goals, specific desired outcomes can be determined. It should be noted that developing countries may face special chal- lenges in formulating national outcomes. Developing countries may find it difficult to set governmental priorities for some of the reasons referred to earlier, including lack of political will, lack of planning and analytical capacity, or a weak central agency. At the same time, though, every government needs to have goals, and there are ways of building a national consensus and developing the necessary capacity to set pri- orities and determine desired outcomes. This entails launching a par- ticipatory process involving key stakeholders. Donor assistance with institution and capacity building can also help jump-start the techni- cal and analytical process of formulating desired national outcomes. The Importance of Building a Participatory and Consultative Process Involving Main Stakeholders Setting goals in isolation leads to a lack of ownership on the part of When choosing outcomes, the main internal and external stakeholders. Likewise, when choos- do not travel the road ing outcomes, it is crucial to build a participatory and consultative alone. process involving the stakeholders. The participatory process should start with the development of goals and continue with setting out- comes and building an indicator system. (Indicators cannot be simply turned over to technicians, because the political apparatus has to be consulted and has to agree on both goals and indicators. We will elaborate on this in Step 3, setting indicators). The new realities of governance, globalization, aid lending, and citizen expectations require an approach that is consultative, cooper- ative, and committed to consensus building. The voices and views of stakeholders should be actively solicited. Engaging key stakeholders in a participatory manner helps to build consensus and gain a com- mitment to reaching the desired outcomes. Agreeing on Outcomes to Monitor and Evaluate 59 The Overall Process of Setting and Agreeing upon Outcomes You need to know where you are going, why you are going there, and how you will know when you get there. There is a political process involved in setting and agreeing upon desired outcomes. Each part is critical to the success of achieving stakeholder consensus with respect to outcomes. Identify Specific Stakeholder Representatives Who are the key parties involved around an issue area (health, educa- tion, and so forth)? How are they categorized, for example, NGO, gov- ernment, donor? Whose interests and views are to be given priority? Identify Major Concerns of Stakeholder Groups Use information gathering techniques such as brainstorming, focus groups, surveys, and interviews to discover the interests of the in- volved groups. Numerous voices must be heard--not just the loudest, richest, or most well-connected. People must be brought into the process to enhance and support a democratic public sector. Translate Problems into Statements of Possible Outcome Improvements It should be noted that formulating problems as positive outcomes is quite different from a simple reiteration of the problem. An outcome- oriented statement enables one to identify the road and destination ahead. We encourage outcomes to be framed positively rather than negatively (figure 2.2). Stakeholders will respond and rally better to positive statements, for example, "We want improved health for in- fants and children," rather than "We want fewer infants and children to become ill." Positive statements to which stakeholders can aspire seem to carry more legitimacy. It is easier to gather a political consen- sus by speaking positively to the desired outcomes of stakeholders. Disaggregate to Capture Key Desired Outcome Outcomes should be disaggregated sufficiently to capture only one improvement area in each outcome statement. A sample outcome might be to "increase the percentage of employed people." To know whether this outcome has been achieved, the goal needs to be disag- gregated to answer the following: 60 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 2.2 Developing Outcome Statements Reformulate the concerns identified by stakeholders into positive, desirable outcomes: From To Rural crops are spoil- Improve farmers' ing before getting to access to markets the market Create incentives Children are drop- for families to keep ping out of school children in school No longer safe to Improve community go out after dark safety · For whom? · Where? · How much? · By when? We need to disaggregate this outcome by examining increased employment in terms of a target group, sector, percentage change, and timeframe. For instance, the disaggregated outcome may be to "increase employment among youth in the rural sector by 20 percent over the next four years." Only by disaggregating the outcome and ar- ticulating the details will we know if we have successfully achieved it. Simplifying and distilling outcomes at this point also eliminates complications later when we start to build a system of indicators, baselines, and targets by which to monitor and evaluate. By dis- aggregating outcomes into subcomponents, we can set indicators to measure results. Develop a Plan to Assess How a Government or Organization Will Achieve These Outcomes When one monitors using the traditional implementation-based tools of inputs, activities, and outputs, the need to be clear about outcomes is much less apparent. Managers would gather inputs, assign activities, Agreeing on Outcomes to Monitor and Evaluate 61 and wait for outputs. But the shortcoming of this approach is that It is best to first reach an completing all of the activities and outputs is not the same thing as agreement on strategic achieving the desired outcomes. The sum of all activities may or may priorities and outcomes, not mean that desired outcomes have resulted. A list of tasks and ac- and then use them to drive tivities does not measure results. Even if all activities were completed resource allocations and within a given timeframe, the desired outcome has not necessarily activities. been achieved. This is not to say that activities are unimportant. The actions needed to manage and implement programs, use resources, and de- liver government services are crucial to the process. They are neces- Being busy is not the same sary--just not sufficient. thing as attaining results. Examples and Possible Approaches What is involved in the actual process of choosing outcomes? The example below illustrates one scenario that may be helpful. Situation After broadly based consultations with key stakeholders, a president has set some important national and sector goals for inclusion in a five-year economic develop- ment plan. The prime minister has in turn been asked by the president to translate these goals into a set of outcomes that can be achieved--and demonstrate progress toward the strategic vision. Actions The prime minister asks the Minister of Finance to lead a 10-week effort to iden- tify desired outcomes. The Minister of Finance forms a task group that includes representatives of the country's stakeholder groups. Stakeholders included Government, civil society, donors. Reason included To build consensus for the process. Three key The finance minister gives the new task responsibilities group three key responsibilities: (a) to identify specific stakeholder representa- tives; (b) to identify major concerns of each stakeholder group; and (c) to trans- late the list of concerns into a list of posi- tive and desirable outcomes to achieve. 62 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 2.3 Outcome Statements Derived from Identified Problems or Issues Policy Area: Education From To School buildings Improve school are not maintained structures to meet and are made from standards of poor materials. market economy. Many children of Rural children gain rural families are un- equal access to able to travel long educational distances to school. services. Schools are not Improved curricula teaching our youth meet market-based the content they economy standards. need for the mar- ket economy. The poor and Children most in vulnerable are need are receiving falling behind and educational not getting a assistance. decent education. Translating problems into positive outcome statements is critical to the process. One must begin with the problems in a given country, then reformulate these concerns into a set of desirable outcomes. In other words, issues and problems need to be recast into a set of solutions. Fig- ures 2.3 and 2.4 provide practical examples to illustrate the process, both correctly (figure 2.3) and incorrectly (figure 2.4). Now consider the importance of capturing only a single outcome in each outcome statement. (This will become critical when we turn to indicators later in Step 3.) Figure 2.4 contains four examples of how NOT to construct outcome statements. The statements list mul- Agreeing on Outcomes to Monitor and Evaluate 63 Figure 2.4 How NOT to Construct Outcome Statements Policy Area: Education From To School buildings Improve school are not maintained structures and aca- and are made from demic standards to poor materials. meet requirements of market economy. Many children of Rural children gain rural families are un- equal access to able to travel long educational and distances to school. medical services. Schools are not Improved curricula teaching our youth and facilities meet the content they market-based need for the mar- economy standards. ket economy. The poor and Children most in vulnerable are need are receiving falling behind and educational and not getting a nutritional decent education. assistance. tiple areas for improvement, complicating the later process of setting indicators. In the examples in figure 2.4 there should be two separate out- come statements while presently they are combined. The first, for example, should read "improve school structures to meet require- ments of market economy," and the second: "improve academic standards to meet requirements of market economy." Likewise, the second statement also contains two outcomes, and should read in- stead as "rural children gain equal access to educational services," and "rural children gain access to medical services." The third state- 64 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 2.5 Developing Outcomes for One Policy Area Example: Education Outcomes Indicators Baselines Targets 1. Nation's children have better access to preschool programs 2. Primary school learning outcomes for children are improved ment should contain two outcomes: "improve curricula to meet mar- ket-based standards," and: "improve facilities to meet market-based standards." Finally, the fourth statement can also be translated into two outcomes: "children most in need are receiving educational assistance," and: "children most in need are receiving nutritional assistance." Choosing outcomes is the first step in building the performance matrix. Figure 2.5 provides examples of possible educational development outcomes. Indicators, baselines, and targets will all flow from this initial step of establishing outcomes. As we move through the steps of the model in subsequent chapters, we will look at how to set indicators, baselines, and targets. We have examined the critical importance of setting outcomes, the issues involved in choosing outcomes to monitor and evaluate, and the importance of building a participatory and consultative political process that includes the main stakeholders. We have identified the sequence of steps for setting outcomes, along with some guidelines for developing outcome statements that can be measured through a set of indicators. We turn next to Step 3, selecting key performance indicators to monitor outcomes. Chapter 3 Step 3: Selecting Key Performance Indicators to Monitor Outcomes Figure 3.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 33 4 5 6 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization How will we know when we have achieved our desired outcomes? After examining the importance of setting achievable and well-de- fined outcomes, and the issues and process involved in agreeing upon those outcomes, we turn next to the selection of key indicators (fig- ure 3.1). Outcome indicators are not the same as outcomes. Indica- tors are the quantitative or qualitative variables that provide a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of an organization against the stated outcome. Indicators should be devel- oped for all levels of the results-based M&E system, meaning that in- dicators are needed to monitor progress with respect to inputs, activi- ties, outputs, outcomes, and goals. Progress needs to be monitored at all levels of the system to provide feedback on areas of success and areas in which improvement may be required. Outcome indicators help to answer two fundamental questions: "How will we know success or achievement when we see it? Are we moving toward achieving our desired outcomes?" These are the ques- 65 66 Ten Steps to a Results-Based Monitoring and Evaluation System tions that are increasingly being asked of governments and organiza- tions across the globe. Consequently, setting appropriate indicators to answer these questions becomes a critical part of our 10-step model. Developing key indicators to monitor outcomes enables managers to assess the degree to which intended or promised outcomes are being achieved. Indicator development is a core activity in building a results-based M&E system. It drives all subsequent data collection, analysis, and reporting. There are also important political and methodological considerations involved in creating good, effective indicators. This chapter specifically considers: (a) indicators required for all levels of the results-based M&E system; (b) translating outcomes into outcome indicators; (c) the "CREAM" of good performance indica- tors; (d) the use of proxy indicators; (e) the pros and cons of using predesigned indicators; (f) constructing indicators and tracking per- formance information; and (g) setting indicators using experience from developing countries. Indicators Are Required for All Levels of Results-Based M&E Systems Setting indicators to measure progress in inputs, activities, outputs, outcomes, and goals is important in providing necessary feedback to the management system. It will help managers identify those parts of an organization or government that may, or may not, be achieving results as planned. By measuring performance indicators on a regular, determined basis, managers and decisionmakers can find out whether projects, programs, and policies are on track, off track, or even doing better than expected against the targets set for performance. This provides an opportunity to make adjustments, correct course, and gain valuable institutional and project, program, or policy experience and knowledge. Ultimately, of course, it increases the likelihood of achieving the desired outcomes. Translating Outcomes into Outcome Indicators When we consider measuring "results," we mean measuring out- comes, rather than only inputs and outputs. However, we must trans- late these outcomes into a set of measurable performance indicators. It is through the regular measurement of key performance indicators that we can determine if outcomes are being achieved. Selecting Key Performance Indicators to Monitor Outcomes 67 For example, in the case of the outcome "to improve student learning," an outcome indicator regarding students might be the change in student scores on school achievement tests. If students are continually improving scores on achievement tests, it is assumed that their overall learning outcomes have also improved. Another ex- ample is the outcome "reduce at-risk behavior of those at high risk of contracting HIV/AIDS." Several direct indicators might be the meas- urement of different risky behaviors for those individuals most at risk. As with agreeing on outcomes, the interests of multiple stakehold- ers should also be taken into account when selecting indicators. We previously pointed out that outcomes need to be translated into a set of measurable performance indicators. Yet how do we know which indicators to select? The selection process should be guided by the knowledge that the concerns of interested stakeholders must be con- sidered and included. It is up to managers to distill stakeholder inter- ests into good, usable performance indicators. Thus, outcomes should be disaggregated to make sure that indicators are relevant across the concerns of multiple stakeholder groups--and not just a single stakeholder group. Just as important, the indicators have to be relevant to the managers, because the focus of such a system is on performance and its improvement. If the outcome is to improve student learning, then one direct stakeholder group is, of course, students. However, in setting up a re- sults system to measure learning, education officials and governments might also be interested in measuring indicators relevant to the con- cerns of teachers and parents, as well as student access to schools and learning materials. Thus, additional indicators might be the number of qualified teachers, awareness by parents of the importance of en- rolling girls in school, or access to appropriate curriculum materials. This is not to suggest that there must be an indicator for every stakeholder group. Indicator selection is a complicated process in What is the ideal number which the interests of several relevant stakeholders need to be consid- of indicators for any one ered and reconciled. At a minimum, there should be indicators that outcome? The minimum directly measure the outcome desired. In the case of improving stu- number that answers the dent learning, there must be an indicator for students. Scores on question: "Has the out- achievement tests could be that particular indicator. come been achieved?" With the addition of outcome indicators (figure 3.2), we can ex- pand on the performance framework for educational development outcomes introduced in the previous chapter. 68 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 3.2 Developing a Set of Outcome Indicators for a Policy Area Example: Education Outcomes Indicators Baselines Targets 1. Nation's children 1. Percent of eligible have better access urban children en- to preschool rolled in preschool programs education 2. Percent of eligible rural children enrolled in preschool education 2. Primary school 1. Percent of Grade 6 learning outcomes students scoring 70% for children are or better on improved standardized math and science tests The "CREAM" of Good Performance Indicators The "CREAM" of selecting good performance indicators is essen- tially a set of criteria to aid in developing indicators for a specific project, program, or policy (Schiavo-Campo 1999, p. 85). Perfor- mance indicators should be clear, relevant, economic, adequate, and monitorable. CREAM amounts to an insurance policy, because the more precise and coherent the indicators, the better focused the measurement strategies will be. Clear Precise and unambiguous Relevant Appropriate to the subject at hand Economic Available at a reasonable cost Adequate Provide a sufficient basis to assess performance Monitorable Amenable to independent validation If any one of these five criteria are not met, formal performance indi- cators will suffer and be less useful5. Performance indicators should be as clear, direct, and unambigu- Selecting Key Performance Indicators to Monitor Outcomes 69 ous as possible. Indicators may be qualitative or quantitative. In es- tablishing results-based M&E systems, however, we advocate begin- ning with a simple and quantitatively measurable system rather than inserting qualitatively measured indicators upfront. Quantitative indicators should be reported in terms of a specific number (number, mean, or median) or percentage. "Percents can also be expressed in a variety of ways, e.g., percent that fell into a particu- lar outcome category . . . percent that fell above or below some tar- geted value . . . and percent that fell into particular outcome intervals . . . " (Hatry 1999, p. 63). "Outcome indicators are often expressed as the number or percent (proportion or rate) of something. Pro- grams should consider including both forms. The number of suc- cesses (or failures) in itself does not indicate the rate of success (or failure)--what was not achieved. The percent by itself does not indi- cate the size of the success. Assessing the significance of an outcome typically requires data on both number and percent" (Hatry 1999, p. 60). "Qualitative indicators/targets imply qualitative assessments . . . [that is], compliance with, quality of, extent of and level of . . . .Qual- itative indicators . . . provide insights into changes in institutional processes, attitudes, beliefs, motives and behaviors of individuals" (U.N. Population Fund 2000, p. 7). A qualitative indicator might measure perception, such as the level of empowerment that local gov- ernment officials feel to adequately do their jobs. Qualitative indica- tors might also include a description of a behavior, such as the level of mastery of a newly learned skill. Although there is a role for quali- tative data, it is more time consuming to collect, measure, and distill, especially in the early stages. Furthermore, qualitative indicators are harder to verify because they often involve subjective judgments about circumstances at a given time. Qualitative indicators should be used with caution. Public sector management is not just about documenting perceptions of progress. It is about obtaining objective information on actual progress that will aid managers in making more well-informed strategic decisions, aligning budgets, and managing resources. Actual progress matters because, ultimately, M&E systems will help to provide information back to politicians, ministers, and organizations on what they can re- alistically expect to promise and accomplish. Stakeholders, for their part, will be most interested in actual outcomes, and will press to hold managers accountable for progress toward achieving the outcomes. 70 Ten Steps to a Results-Based Monitoring and Evaluation System Performance indicators should be relevant to the desired outcome, and not affected by other issues tangential to the outcome. The economic cost of setting indicators should be considered. This Every indicator has cost means that indicators should be set with an understanding of the and work implications. In likely expense of collecting and analyzing the data. essence, when we explore For example, in the National Poverty Reduction Strategy Paper building M&E systems, we (PRSP) for the Kyrgyz Republic, there are about 100 national and are considering a new subnational indicators spanning more than a dozen policy reform M&E system for every areas. Because every indicator involves data collection, reporting, and single indicator. Therefore, analysis, the Kyrgyz government will need to design and build 100 indicators should be chosen individual M&E systems just to assess progress toward its poverty carefully and judiciously. reduction strategy. For a poor country with limited resources, this will take some doing. Likewise, in Bolivia the PRSP initially con- tained 157 national-level indicators. It soon became apparent that building an M&E system to track so many indicators could not be sustained. The present PRSP draft for Bolivia now has 17 national- level indicators. Indicators ought to be adequate. They should not be too indirect, too much of a proxy, or so abstract that assessing performance be- comes complicated and problematic. Indicators should be monitorable, meaning that they can be inde- pendently validated or verified, which is another argument in favor of starting with quantitative indicators as opposed to qualitative ones. Indicators should be reliable and valid to ensure that what is being measured at one time is what is also measured at a later time-- and that what is measured is actually what is intended. Caution should also be exercised in setting indicators according to the ease with which data can be collected. "Too often, agencies base their selection of indicators on how readily available the data are, not how important the outcome indicator is in measuring the extent to which the outcomes sought are being achieved" (Hatry 1999, p. 55). Figure 3.3 is an additional checklist for assessing proposed indicators. The Use of Proxy Indicators "Better to be approximately You may not always be precise with indicators, but you can strive to correct than precisely be approximately right. Sometimes it is difficult to measure the out- wrong." come indicator directly, so proxy indicators are needed. Indirect, or proxy, indicators should be used only when data for direct indicators (Anon.) are not available, when data collection will be too costly, or if it is Selecting Key Performance Indicators to Monitor Outcomes 71 Figure 3.3 Checklist for Assessing Proposed Indicators Outcome to be measured: __________________________________ Indicator selected: ________________________________________ Is the indicator . . . 1. As direct as possible a reflection of the outcome itself? _____ 2. Sufficiently precise to ensure objective measurement? _____ 3. Calling for the most practical, cost-effective collection of data? _____ 4. Sensitive to change in the outcome, but relatively unaffected by other changes? _____ 5. Disaggregated as needed when reporting on the outcome? _____ Source: United Way of America 1996. not feasible to collect data at regular intervals. However, caution should be exercised in using proxy indicators, because there has to be a presumption that the proxy indicator is giving at least approximate evidence on performance (box 3.1). For example, if it is difficult to conduct periodic household surveys in dangerous housing areas, one could use the number of tin roofs or television antennas as a proxy measure of increased household in- Box 3.1 Indicator Dilemmas The Chicago Museum of Science and Industry--a large, cavernous mu- seum with many monumental-size exhibits, including an entire submarine and a coal mine--wanted to conduct a study to determine which exhibi- tions were of greatest interest to its visitors. They found that it was impos- sible to count how many visitors viewed every exhibit, so they decided to use a proxy indicator. They did this by determining where they needed to replace floor tiles most often. And where did they find the floor tiles most in need of replacement? In front of the exhibit of hatching baby chicks. Source: Webb et al., 1966. 72 Ten Steps to a Results-Based Monitoring and Evaluation System come. These proxy indicators might be correctly tracking the desired outcome, but there could be other contributing factors as well; for example, the increase in income could be attributable to drug money, or income generated from the hidden market, or recent electrification that now allows the purchase of televisions. These factors would make attribution to the policy or program of economic development more difficult to assert. The Pros and Cons of Using Predesigned Indicators Predesigned indicators are those indicators established independently of an individual country, organization, program, or sector context. For example, a number of development institutions have created in- dicators to track development goals, including the following: · MDGs · The United Nations Development Programme's (UNDP's) Sustainable Human Development goals · The World Bank's Rural Development Handbook · The International Monetary Fund's (IMF's) Financial Soundness Indicators. The MDGs contain eight goals, with attendant targets and indica- tors assigned to each. For example, Goal 4 is to reduce child mortal- ity, while the target is to reduce by two-thirds the under-five mortal- ity rate between the years 1990 and 2015. Indicators include (a) under-five mortality rate; (b) infant mortality rate; and (c) proportion of one-year-old children immunized against measles. (For a complete list of MDG indicators, see annex 3.) The UNDP created the Human Development Index (HDI) in 1990 as a way of measuring human progress and the quality of life in all countries of the world. "The HDI constitutes the first comprehensive attempt to measure achievements in development from a human per- spective, expressed in terms of numerical indicators that permit inter- country and inter-temporal comparisons . . . The index also provides an initial working tool that could be further developed and refined, and that could guide country efforts to establish relevant databases" (UNDP 2001). More specifically, "[t]he UNDP's Human Development Index measures a country's achievements in three aspects of human devel- Selecting Key Performance Indicators to Monitor Outcomes 73 opment: longevity, knowledge, and a decent standard of living. Longevity is measured by life expectancy at birth; knowledge is measured by a combination of the adult literacy rate and the com- bined gross primary, secondary, and tertiary enrollment ratio; and standard of living, as measured by GDP per capita" (UNDP 2001). The World Bank's Rural Development Indicators Handbook, based on the World Development Indicators, defines and dissemi- nates international statistics on a broad set of rural indicators for rural well-being, improvement in the rural economy, development of rural markets, improvement of accessibility and communication, sus- tainable management of the resource base, and policy and institu- tional framework. Specific indicators include, for example, rural population below the poverty line, agricultural gross domestic prod- uct, agricultural exports, paved roads, potential arable land, and local tax revenue. Thus, the Rural Development Indicators Handbook helps to de- velop a common approach to monitoring and evaluating progress both within and across countries using a common, clearly defined set of indicators. The Handbook also contains a Rural Score Card--a composite indicator that can be used, for example, to assess a coun- try's overall progress (or lack thereof) toward achievement of rural poverty reduction (World Bank 2000). In light of regional financial crises in various parts of the world, the IMF is in the process of devising a set of Financial Soundness In- dicators. These are indicators of the current financial health and soundness of a given country's financial institutions, corporations, and households. They include indicators of capital adequacy, asset quality, earnings and profitability, liquidity, and sensitivity to market risk (IMF 2003). On a more general level, the IMF also monitors and publishes a se- ries of macroeconomic indicators that may be useful to governments and organizations. These include output indicators, fiscal and mone- tary indicators, balance of payments, external debt indicators, and the like. There are a number of pros and cons associated with using pre- designed indicators: Pros: · They can be aggregated across similar projects, programs, and policies. 74 Ten Steps to a Results-Based Monitoring and Evaluation System · They reduce costs of building multiple unique measurement systems. · They make possible greater harmonization of donor requirements. Cons: · They often do not address country specific goals. · They are often viewed as imposed, as coming from the top down. · They do not promote key stakeholder participation and ownership. · They can lead to the adoption of multiple competing indicators. There are difficulties in deciding on what criteria to employ when one chooses one set of predesigned indicators over another. Predesigned indicators may not be relevant to a given country or organizational context. There may be pressure from external stake- holders to adopt predesigned indicators, but it is our view that indi- cators should be internally driven and tailored to the needs of the or- ganization and to the information requirements of the managers, to the extent possible. For example, many countries will have to use some predesigned indicators to address the MDGs, but each country should then disaggregate those goals to be appropriate to their own particular strategic objectives and the information needs of the rele- vant sectors. Ideally, it is best to develop indicators to meet specific needs while involving stakeholders in a participatory process. Using predesigned in- dicators can easily work against this important participatory element. Constructing Indicators Constructing indicators takes work. It is especially important that It will take more than one competent technical, substantive, and policy experts participate in try to develop good indica- the process of indicator construction. All perspectives need to be tors. Arriving at a final set taken into account--substantive, technical, and policy--when con- of appropriate indicators sidering indicators. Are the indicators substantively feasible, techni- will take time. cally doable, and policy relevant? Going back to the example of an outcome that aims to improve student learning, it is very important to make sure that education professionals, technical people who can construct learning indicators, and policy experts who can vouch for the policy relevance of the indicators, are all included in the discus- sion about which indicators should be selected. Indicators should be constructed to meet specific needs. They also need to be a direct reflection of the outcome itself. And over time, Selecting Key Performance Indicators to Monitor Outcomes 75 new indicators will probably be adopted and others dropped. This is to be expected. However, caution should be used in dropping or mod- ifying indicators until at least three measurement have been taken. Taking at least three measurements helps establish a baseline and a trend over time. Two important questions should be answered before changing or dropping an indicator: Have we tested this indicator thoroughly enough to know whether it is providing information to effectively measure against the desired outcome? Is this indicator pro- viding information that makes it useful as a management tool? It should also be noted that in changing indicators, baselines against which to measure progress are also changing. Each new indi- cator needs to have its own baseline established the first time data are collected for it. (The topic of setting baselines is covered in fur- ther detail in chapter 4.) In summary, indicators should be well thought through. They should not be changed or switched often (and never on a whim), as this can lead to chaos in the overall data collection system. There should be clarity and agreement in the M&E system on the logic and rationale for each indicator from top level decisionmakers on to those responsible for collecting data in the field. Performance indicators can and should be used to monitor out- comes and provide continuous feedback and streams of data "The central function throughout the project, program, or policy cycle. In addition to using of any performance meas- indicators to monitor inputs, activities, outputs, and outcomes, indi- urement process is to pro- cators can yield a wealth of performance information about the vide regular, valid data on process of and progress toward achieving these outcomes. Informa- indicators of performance tion from indicators can help to alert managers to performance dis- outcomes." crepancies, shortfalls in reaching targets, and other variabilities or (Hatry 1999, p. 17) deviations from the desired outcome. Thus, indicators provide organizations and governments with the opportunity to make midcourse corrections, as appropriate, to man- age toward the desired outcomes. Using indicators to track process and progress is yet another demonstration of the ways that a results- based M&E system can be a powerful public management tool. Setting Indicators: Experience in Developing Countries More and more developing countries--and even regions--are begin- ning to set indicators to track progress toward their development goals. Boxes 3.2 through 3.4 review experiences in the Africa region, Sri Lanka, and Albania. 76 Ten Steps to a Results-Based Monitoring and Evaluation System Box 3.2 The Africa Region's Core Welfare Indicators Efforts are underway throughout the Africa region to create the basic statistical and technical building blocks of M&E systems. Among these building blocks are the core indicators surveys that have been con- ducted in a number of African countries, including Ghana, Malawi, Mozambique, Nigeria, and Lesotho. The Core Welfare Indicators Questionnaire (CWIQ) was created jointly by the World Bank, the UNDP, and UNICEF to monitor development objectives through the use of leading indicators in general, and social in- dicators in particular. "Leading indicators are indicators which give advance warning of a future impact, whose emergence may be delayed or difficult to measure"(http://www4.worldbank.org/afr/stats/pdf/cwiq.pdf). Specifically, the CWIQ helps governments collect indicators related to household well-being, and indica- tors of access to, usage of, and satisfaction with basic services on an annual basis. CWIQ features include the following: · A fixed set of core questions with flexible modules · Quick data entry and validation · Simple reporting · Large sample · Short questionnaire · Easy data collection. "The CWIQ is not a complicated survey. It incorporates a package of features, which, when taken together, ensure wide coverage and a rapid turnaround time" (www.worldbank.org/afr/stats/pdf/ghcoreinds.pdf). The CWIQ also " . . . provides key social indicators for different population subgroups--within and across countries; [acts as] . . . an instrument for monitoring changes in key social indicators over time; and provides countries with a simple tool that produces rapid results" (World Bank p. 1). At the same time, using the CWIQ does not prohibit in any way participant countries from also develop- ing their own specific socioeconomic indicators. For an example of a completed CWIQ, go to http://www.worldbank.org/afr/stats/pdf/ghcoreinds.pdf, which contains the Core Welfare Indicators for Ghana (http://www4.worldbank.org/afr/stats/ pdf/cwiqloop.pdf). Source: World Bank. Selecting Key Performance Indicators to Monitor Outcomes 77 Box 3.3 Sri Lanka's National Evaluation Policy The government of Sri Lanka's National Evaluation Policy seeks to: (a) create an evaluation culture and to use evaluations to manage for results; (b) promote evaluation through capacity building with respect to staff, institutions, tools, and methodologies; (c) enable learning of lessons from past experiences; (d) im- prove the design of development policies and programs through integration of evaluating findings; and (e) establish accountability, transparency, and good governance. As part of the evaluation policy, the government is mandating the use of performance indicators for all policy, program, and project preparation initiatives. For this purpose, the government is encouraging part- nerships with civil society organizations (for example, the Sri Lanka Evaluation Association) and NGOs to introduce participatory evaluations in the public sector. The government is also encouraging universities and public sector training institutions to include evaluation modules to share knowledge on evaluation techniques and methodologies. Also see annex 4: The Sri Lanka National Development Plan for Monitoring and Evaluation. Source: Sri Lanka Evaluation Association and Ministry of Public Development and Implementation 2003. To the extent possible, indicators should be developed based on the particular needs of a given country or organization. " . . . [T]he appropriate choice of performance indicators differ for different countries, times, and sectors. The only valid general rule is, therefore, when performance measurement is appropriate and cost-effective, performance should be assessed according to that combination of output, outcome and process indicators that are realistic and suitable for the specific activity, sector, country, and time" (Schiavo-Campo 1999, pp. 80­81). Again, developing good indicators inevitably takes more than one try, and arriving at the final set of indicators will take time. What we are ultimately building is a performance framework to provide countries and organizations with the means to develop strategies, set outcomes, build indicators, establish baselines, and set targets. This process will help guide the best use of budgets, re- sources, and personnel to achieve the desired outcomes. 78 Ten Steps to a Results-Based Monitoring and Evaluation System Box 3.4 Albania's Three-Year Action Plan Monitoring and evaluation systems--both implementation and performance-based--will be developed and used by the government of Albania to provide feedback on major programs constituting the Three-Year Action Plan (including all major strategic initiatives currently underway in Albania's public sector: the National Strategy for Social and Economic Development [NSSED]; the Medium-Term Expenditure Frame- work; the Stabilization and Association Agreement; the Anti-Corruption Action Plan; and the Strategy for Decentralization and Local Autonomy). The government has assigned the Coordination Department within the Council of Ministers to oversee and coordinate implementation monitoring of the Three-Year Action Plan. Similar responsibilities for NSSED performance monitoring will be assigned to the NSSED Depart- ment within the Ministry of Finance. The Ministry of Finance is expected to oversee the overall performance and implementation manage- ment by the 12 line ministries covered by the NSSED. Responsibilities include: (a) procedures for setting indicators that will be tracked and reported on; (b) instructions to the line ministries on how to select indi- cators; (c) processes for selecting indicators to ensure they measure results that key stakeholders care about; and (d) procedures clarifying how information is to be collected against the indicators to ensure verification and reporting consistency. Progress is also being made in the Education Ministry, which recently developed a draft NSSED progress monitoring matrix. A new M&E unit has also been established within the Education Ministry, including six representatives of different departments. A variety of education indicators will be developed in connec- tion with the government's Growth and Poverty Reduction Strategy, Poverty Reduction Support Credit, Education Project, and Education for All, initiatives. Education indicators include, among others, school attendance by educational level, teacher salaries, share of GDP spent on education, pupil-teacher ratio, per- centage of the teaching force that meets ministry standards for qualified teachers, average class size, educa- tion completion rates overall, and education rates disaggregated for rural and poor families. More generally, the Albanian government has basic statistical capacity (although there is room for im- provement), and recently established a policy analysis unit. The government also has the indicators in place with respect to the MDGs. Source: World Bank 2002a. The following are examples of indicators at various levels: Box 3.5 provides some useful examples of program and project level indicators. Box 3.6 provides an example of an outcome and some possible indicators. Selecting Key Performance Indicators to Monitor Outcomes 79 Box 3.5 Program and Project Level Results Indicators: An Example from the Irrigation Sector Project name Strengthening irrigation in a specific country area Project goals Improve agricultural productivity Raise farm income. Indicators Outcome indicators New area under irrigation Higher yield Increased production Increased farm income. Output indicators Construction of 10 new irrigation schemes Reconstruction of five old irrigation schemes Twenty-five farmer training sessions. Source: Adapted from IFAD 2002, p.19. Box 3.6 Outcome: Increased Participation of Farmers in Local Markets Possible outcome indicators · Percent change in annual revenue · Percent change in amount of spoiled crops · Percent change in crop pricing due to competition · Percent change in agricultural employment. Chapter 4 Step 4: Setting Baselines and Gathering Data on Indicators Figure 4.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 44 5 6 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization After working through the process of selecting key performance indi- cators to monitor outcomes, we turn next to Step 4 and the establish- ment of baseline data, that is, establishing where we are at present relative to the outcome we are trying to achieve (figure 4.1). One cannot project performance into the future (set targets) without first establishing a baseline. The baseline is the first measurement of an in- dicator. It sets the current condition against which future change can be tracked. For instance, it helps to inform decisionmakers about current circumstances before embarking on projecting targets for a given program, policy, or project. In this way, the baseline is used to learn about current or recent levels and patterns of performance. Im- portantly, baselines provide the evidence by which decisionmakers are able to measure subsequent policy, program, or project performance. This chapter specifically covers: (a) establishing baseline data on indicators; (b) building baseline information; (c) identifying data sources for indicators; (d) designing and comparing data collection methods; (e) the importance of conducting pilots; and (f) data collec- tion, using some developing country experiences. 80 Setting Baselines and Gathering Data on Indicators 81 Establishing Baseline Data on Indicators Establishing baselines is the third part of the performance frame- work. Baselines are derived from outcomes and indicators. We would note in beginning this examination of baselines that es- tablishing baselines is not an exotic idea. We gauge our personal per- formance against our own baseline data in our own lives. For ex- ample, we check our blood pressure against what we have had at one time in the past, track our capacity to exercise against our perform- ance when we first began to exercise, and keep an eye on our weight against an earlier weight. A performance baseline is information--qualitative or quantita- tive--that provides data at the beginning of, or just prior to, the monitoring period. The baseline is used as a starting point, or guide, by which to monitor future performance. Baselines are the first criti- cal measurement of the indicators. Figure 4.2 contains an example of baseline data for a particular policy area. It builds on the performance framework introduced in Figure 4.2 Developing Baseline Data for One Policy Area Example: Education Outcomes Indicators Baselines Targets 1. Nation's children 1. Percent of eligible 1. In 1999, 75 percent have better access urban children en- of children ages 3­5 to preschool rolled in preschool 2. In 2000, 40 percent programs education of children ages 3­5 2. Percent of eligible rural children enrolled in preschool education 2. Primary school 1. Percent of Grade 6 1. In 2002, 75 percent learning outcomes students scoring 70% scored 70 percent or for children are or better on better in math, and improved standardized math 61 percent scored and science tests 70 percent or better in science 82 Ten Steps to a Results-Based Monitoring and Evaluation System chapter 1. (We will complete the framework when we discuss Step 5, Setting Targets.) The challenge is to obtain adequate baseline information on each of the performance indicators for each outcome. This can quickly be- come a complex process. It is important to be judicious in the num- ber of indicators chosen, because each indicator will need data collec- tion, analysis, and reporting systems behind it. Building Baseline Information There are eight key questions that should be asked in building base- line information for every indicator. (These questions continue to apply in subsequent efforts to measure the indicator.) 1. What are the sources of data? 2. What are the data collection methods? 3. Who will collect the data? 4. How often will the data be collected? 5. What is the cost and difficulty to collect the data? 6. Who will analyze the data? 7. Who will report the data? 8. Who will use the data? So, for each indicator, we will need to complete table 4.1. The statistical systems in developed countries frequently can de- liver precise information for all three stages of traditional implemen- tation monitoring--inputs, activities, and outputs. However, develop- Table 4.1 Building Baseline Information Who Cost and Who Who Who Data will difficulty will will will Data collection collect Frequency to analyze report use Indicator source method data? to collect collect data? data? data? 1 2 3 Setting Baselines and Gathering Data on Indicators 83 ing countries generally have less sophisticated systems. The data sys- tems may not be available and may vary with respect to precision. Some countries will know with reasonable precision how many rural children are in school, while others will have only rough estimates. Other developing countries may know the utilization rates of hospi- tal beds, and some may not. The selected performance indicators, and the data collection strate- gies used to track those indicators, need to be grounded in the reali- ties of what data systems are in place, what data can presently be produced, and what capacity exists to expand the breadth and depth of data collection and analysis. Identifying Data Sources for Indicators Every indicator constitutes its own miniature M&E system, so the first consideration in starting to build the information system for that indicator is what sources of information potentially can supply the relevant data. A number of issues need to be considered when identifying data sources. Can the data source be accessed in a practical fashion? Can Sources are who or what the data source provide quality data? Can the data source be ac- provide data--not the cessed on a regular and timely basis? Is primary data collection from method of collecting data. the information source feasible and cost effective? It is important to collect only the data that is intended to be used. After all, performance information should be a management tool-- and there is no need to collect information that managers are not going to use. "As a rule of thumb, only collect baseline information that relates directly to the performance questions and indicators that you have identified. Do not spend time collecting other information" (IFAD 2002, Section 5, p. 32). Data sources for indicators can be primary or secondary. Primary data are collected directly by the organization concerned, and may include administrative, budget, or personnel data; surveys; inter- views; and direct observation. Secondary data have been collected by other outside organizations, and are gathered for purposes other than those of the organization concerned. Examples of secondary data in- clude survey data collected by another agency (UNDP or UNESCO [United Nations Educational, Scientific and Cultural Organization], for example), financial market data, or demographic health survey data. There are pros and cons associated with the use of secondary data 84 Ten Steps to a Results-Based Monitoring and Evaluation System to establish performance trends on indicators. On the positive side, secondary data can be more cost efficient. Secondary data may also be used in instances when it is not practical or possible to collect pri- mary data frequently, as in the case of large scale and expensive household surveys. However, for a variety of reasons, secondary data must be used with caution. Secondary data will have been gathered with other or- ganization goals or agendas in mind. Other questions arise in using secondary data as well: Are the data valid? Are they reliable? How often are the data collection instruments validated? Furthermore, using secondary data means using someone else's data to report progress and success in moving toward your own desired outcomes. Are you as a manager comfortable with this arrangement, given all the advantages and disadvantages of doing so? Examples of sources of actual data may include administrative records (written or computerized) from government and nongovern- ment organizations; interviews and surveys with client target groups, program officials, and service providers; reports from trained ob- servers; and mechanical measurements and tests. An increasing understanding of the need for streams of informa- tion, not discrete studies that are episodic and spaced out over time, is emerging in public sector organizations throughout the world. Managers are looking for information--whether on policy strategies, utilization of health clinics, farming methods, or migration pat- terns--that they can trust and use in real time. Waiting for months or even a year or more for studies to be completed is not helpful. The new approach to building results-based M&E systems is increasingly toward building those systems that provide more or less continuous information streams. Designing and Comparing Data Collection Methods Over time, internal If the sources of data are known, what will be the strategies and in- organizational capacity struments for data collection? Decisions will need to be made regard- for data collection and ing how to obtain the necessary data from each source, how to pre- analysis can and should be pare the data collection instruments to record the information built, as it is a key compo- appropriately, what procedures to use (surveys versus interviews, for nent in establishing a sus- example), how often to access the data sources, and so forth. tainable M&E system. The government might also contract externally to use existing ca- pacity at universities and research centers for data collection efforts. Data collection can also be purchased from private sector providers. Setting Baselines and Gathering Data on Indicators 85 Figure 4.3 Data Collection Methods Panel surveys Key Conversation informant Reviews of with interviews official One-time concerned records survey individuals (management Participant information Direct observation system and observation Community administra- interviews Focus tive data) group interviews Census Field Field visits Questionnaires experiments Informal and less-structured methods Formal and more-structured methods Source: Adapted from Marchant 2000. However, any strategy that involves the long-term purchase of data collection from nongovernment vendors has certain vulnerabilities and is likely to be more expensive. Figure 4.3 illustrates some of the possible methods of collecting data. There is no correct answer as to which method is best. It will depend on a given organization's resource availability, access, needs, time constraints, and so forth. It will also depend on the needs of the user of the information. For example, there may be questions about how much precision is actually needed by a given user in light of tradeoffs of cost and time. A combination of data collection strategies might work best in building the information system to support tracking each indicator. For example, an organization could choose to have only a few indi- cators and draw on data collection strategies from different places along the continuum. There is no one right approach to the selection of data collection strategies. A number of contingencies help to frame what is possible and what can be afforded. It is worth some time to understand the implications of choosing one collection strategy in comparison to other options. To just decide in an ad hoc, off-hand way to use surveys, or to conduct multiple 86 Ten Steps to a Results-Based Monitoring and Evaluation System focus groups, or to undertake a household survey, is to create possi- bly critical problems later on. Table 4.2 is an illustrative comparison of four major data collec- tion methods along four dimensions. It highlights some of the trade- offs among different strategies. Before any decisions are made on the strategies to deploy, it is important to check with the users. Try and determine their level of comfort with the tradeoffs and with the sorts of performance information they will be receiving. Data collection strategies necessarily involve some tradeoffs with respect to cost, precision, credibility, and timeliness. For example, the more structured and formal methods for collecting data generally tend to be more precise, costly, and time consuming. If data are needed frequently and on a routine basis to inform management deci- sionmaking, it may be preferable to adopt less precise, more unstruc- tured, and inexpensive data collection strategies. From the beginning, we have noted that the 10-step model in this handbook is not strictly linear and sequential. As they build perform- ance systems, organizations will need to go back and forth among the steps. The development and fine tuning of the system will con- tinue, and the information needs of users will change--requiring new indicators, new baseline data, and so forth. The result is that there needs to be a certain degree of adaptability and flexibility in the sys- tem to identify new data sources, new collection techniques, and new ways of reporting. The Importance of Conducting Pilots Piloting of indicators and the information requirements behind them should be done--period. It is extremely risky to move to full imple- mentation of an indicator system at any level in a government, or even an individual organization, before thorough testing of the data sources, collection and analysis strategies, and means of reporting. The pilot is a means of learning what works and what does not. It is a way of making small mistakes early rather than big mistakes later. A pilot alerts managers that there are some indicators for which data do not exist, or for which data are too costly, time consuming, or complex to obtain. This is crucial information to have as the base- line is established. The pilot might demonstrate that it would be eas- ier to set an indicator on the basis of existing secondary data that are already being collected across an organization or government as op- posed to creating a new indicator that needs its own M&E system. Setting Baselines and Gathering Data on Indicators 87 Table 4.2 Comparison of Major Data Collection Methods Data collection method Review of Self- Rating by program administered trained Characteristic records questionnaire Interview observer Cost Low Moderate Moderate Depends on to high availability of low-cost observers Amount of Some None to some Moderate Moderate training required to high to high for data collectors Completion Depends on Moderate Moderate Short to time amount of moderate data needed Response High, if records Depends on Generally High rate contain needed how distributed moderate to data good Source: United Way of America 1996. The use of existing data systems can be quite helpful in the early stages of building a results-based M&E system. It is important to "[r]ecognize that an existing data collection system may offer a par- tial route, at a minimum, for collecting some of the needed data, pos- sibly at a reduced cost. [There may be] an opportunity . . . for using parts of an existing data set by selecting, adding, or modifying data elements . . . Design a sample--based on an existing data collection system, new collection procedures, or a combination of the two--and extrapolate to the universe" (Wye 2002, p. 31). The pilot is the correct time to step back and look at any proposed indicators as they relate to data collection strategies. If every indica- tor will require costly data collection methods, some rethinking of the indicators is necessary. One should choose indicators that will yield the best information at the lowest cost. This is an opportunity to start to rationalize and prioritize the set of indicators. There will 88 Ten Steps to a Results-Based Monitoring and Evaluation System be continuing pressure from stakeholders to include more indicators, but it is better to have fewer indicators than a multitude of them. For example, the Comprehensive Development Framework in the Kyrgyz Republic mentioned earlier initially included a list of nearly 100 national indicators--each entailing explicit data collection Box 4.1 Albania's Strategy for Strengthening Data Collection Capacity The government of Albania is embarking on a number of policy initiatives, such as the Growth and Poverty Reduction Strategy, that will require good information for policymakers as they move forward in the design, implementation, and evaluation of socioeconomic programs. The government, with the help and support of some international donors, is seeking to improve the country's data practices to produce reliable statistical information on a regular basis to measure and monitor poverty, inequality, and other social indi- cators. Specifically, the project will assist the government in four areas: (a) data collection; (b) data process- ing and analysis; (c) data dissemination and usage; and (d) survey organization and administration. With respect to data collection, the project will provide technical assistance and hands-on training to enhance capacity at the Albanian Institute of Statistics (INSTAT). The goal is to help INSTAT to regularly produce a number of surveys, such as the Living Standards Measurement Survey, a Population and House- hold Census, Household Budget Surveys, and annual Labor Force Surveys. Additional work is being planned with several line ministries to do poverty mapping. Regarding data processing and analysis, the project will support improvements in the efficiency and use of information, as well as support for institutional capacity to process household-level information. Technical assistance and hands-on training in the areas of data entry, cleaning, and editing will also be provided to help ensure the quality and timeliness of the information generated. The areas of data analysis include use of sta- tistical and Geographic Information Systems software, poverty analysis methodology, collection and analysis of panel data, household survey techniques, and questionnaire development, sampling, and poverty mapping. Data dissemination and usage will be supported with the aim of fostering a participatory process for generation of statistical information. Capacity building will be directed at both producers and users of sta- tistical household information. A data users group will be formed and will be chaired by INSTAT. The users group will contain representatives from line ministries, donors, and NGOs. A comprehensive strategy will be developed to publish and disseminate results. Finally, survey organization and administration will be supported by the project in the form of a review of INSTAT organization, with a particular focus on those INSTAT units directly engaged in household sur- vey work. The review will assess options for strengthening INSTAT's organizational capacity to manage and administer regular household surveys, and will develop a related staffing plan. The review will also as- sess the internal organizational procedures for administering the survey, including lines of managerial and financial subordination, and will develop a package of related internal administrative procedures. Source: World Bank 2002d. Setting Baselines and Gathering Data on Indicators 89 strategies for measuring them. For each of these indicators, one must Use existing information consider the seven key questions on data collection and management. and data systems whenever Obviously, so many indicators can be difficult to track, and will be a possible--so long as they drain on the resources of a developing country. Reducing the number are trustworthy, fit the in- of indicators is surely preferable in such a case. formation needs, and are Data Collection: Two Developing Country Experiences accessible over time. Boxes 4.1 and 4.2 provide examples of data collection in two devel- oping countries. The government of Albania is working to build ca- pacity and to reform data practices. The government of Lebanon is joining the IMF data system to align its data collection and statistical system with IMF and international standards. Establishing baseline data on indicators is crucial in determining current conditions and in measuring future performance against the starting point. Subsequent and continuous measurements from the baseline will provide important directional or trend data, and can help decisionmakers determine whether they are on track in achieving the desired outcomes over time. But making the decisions on the perform- ance information data to collect, how to collect and analyze it, and how to report it are all important. Pilots can help frame the decisions. Box 4.2 Lebanon: Joining the IMF Data System The government of Lebanon is making an effort to bring its statistical data collection up to international standards. It recently joined the General Data Dissemination System (GDDS) of the IMF. "The purposes of the GDDS are to: encourage member countries to improve data quality; provide a framework for evaluating needs for data improvement and setting priorities . . . ; and guide member countries in the dissemination to the public of comprehensive, timely, accessible, and reliable economic, financial and socio-demographic statistics . . . [It] is built around four dimensions--data characteristics, quality, access and integrity-- and is intended to provide guidance for the overall development of macroeconomic, financial, and socio- demographic data" (IMF 2002). "`Lebanon's membership in the International Monetary Fund's data system is expected to help boost good governance in the country. . . . By selecting the GDDS as a framework to develop the country's national statistical systems, the authorities have underscored their commitment to improving the produc- tion of economic and socio-demographic data . . . this will help increase international recognition of Lebanon's commitment to better statistics,' the director of statistics for the IMF said . . . " (The Daily Star 2003). The statistics will be posted and available to the public in three languages on the Lebanese Central Bank Web site, and will be updated regularly by the Central Bank and the line ministries. Sources: The Daily Star 2003; IMF 2002. Chapter 5 Step 5: Planning for Improvement--Selecting Results Targets Figure 5.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 5 6 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization After gathering baseline data on indicators, the next step is to estab- lish results targets--what can be achieved in a specific time toward reaching the outcome (figure 5.1). Identifying the expected and de- sired level of project, program, or policy results requires the selection of specific performance targets. Target setting is the final step in building the performance frame- work. It, in turn, is based on outcomes, indicators, and baselines. The reasoning process is a deductive one, flowing back from the de- sired outcomes. This chapter will address (a) a definition of targets; (b) factors to consider when selecting indicator targets; (c) examples of targets re- lated to development issues; and (d) the overall performance-based framework. Definition of Targets A target is " . . . a specified objective that indicates the number, timing and location of that which is to be realized"6 (IFAD 2002, 90 Planning for Improvement--Selecting Results Targets 91 Figure 5.2 Identifying Desired Level of Results Requires Selecting Performance Targets Baseline Desired Target indicator level of performance level + improvement = (Desired level (Assumes a of performance finite and to be reached expected level within a of inputs, specific time) activities, and outputs) p. A-11). In essence, targets are the quantifiable levels of the indica- tors that a country, society, or organization wants to achieve by a given time. For example, one target might be "all families should be able to eat two meals a day, every day, by 2005." One method to establish targets is to start with the baseline indica- tor level, and include the desired level of improvement (taking into consideration available resources over a specific time period, for ex- ample, 24­36 months), to arrive at the performance target. In so doing, the starting point will be known, as will the available re- sources to make progress toward that target over a particular period of time. This will give the target performance. The formula in figure 5.2 shows the process for devising per- formance targets. Factors to Consider When Selecting Performance Indicator Targets There are a number of important factors to consider when selecting performance indicator targets. One factor is the importance of taking baselines seriously. There must be a clear understanding of the baseline starting point; for example, an average of the last three years' performance, last year's performance, average trend, data over the past six months, and so forth. In other words, previous 92 Ten Steps to a Results-Based Monitoring and Evaluation System performance should be considered in projecting new performance targets. One might observe how an organization or policy has performed over the previous few years before projecting future performance targets. Another consideration in setting targets is the expected funding "The baseline is the situa- and resource levels--existing capacity, budgets, personnel, funding tion before a program or resources, facilities, and the like--throughout the target period. This activity begins; it is the can include internal funding sources as well as external funding from starting point for results bilateral and multilateral donors. Targets should be feasible given all monitoring. The target is of the resource considerations as well as organizational capacity to what the situation is ex- deliver activities and outputs. pected to be at the end of a Most targets are set annually, but some could be set quarterly. program or activity . . . Others could be set for longer periods. However, setting targets more A thorough analysis of the than three to four years forward is not advisable. There are too many key factors influencing a unknowns and risks with respect to resources and inputs to try to development problem com- project target performance beyond three to four years. In short, be plements the development realistic when setting targets. of baseline data and target The political nature of the process also comes into play. Political setting." concerns are important. What has the government or administration (UNDP 2002, pp. 66­67) promised to deliver? Citizens have voted for a particular government based on articulated priorities and policies that need to be recognized and legitimized in the political process. Setting targets is part of this political process, and there will be political ramifications for either meeting or not meeting targets. Setting realistic targets involves the recognition that most desired Targets are based on outcomes are longer term, complex, and not quickly achieved. Thus, known resources (financial there is a need to establish targets as short-term objectives on the and organizational) plus a path to achieving an outcome. reasonable projection of So how does an organization or country set longer-term, strategic the available resource base goals to be met perhaps 10 to 15 years in the future, when the over a fixed period of time. amount of resources and inputs cannot be known? Most govern- ments and organizations cannot reliably predict what their resource base and inputs will be 10 to 15 years ahead. The answer is to set in- terim targets over shorter periods of time when inputs can be better known or estimated. "Between the baseline and the . . . [outcome] there may be several milestones [interim targets] that correspond to Targets are interim steps expected performance at periodic intervals" (UNDP 2002, p. 66). on the way to an outcome For example, the MDGs have a 15-year time span. While these and eventually to a longer- long-term goals are certainly relevant, the way to reach them is to set term goal. targets for what can reasonably be accomplished over a set of three- Planning for Improvement--Selecting Results Targets 93 to four-year periods. The aim is to align strategies, means, and inputs Each indicator is expected to track progress toward the MDGs over shorter periods of time with to have only one target a set of sequential targets. Targets could be sequenced: target one over a specified time frame. could be for years one to three; target two could be for years four to seven, and so on. Flexibility is important in setting targets because internal or exter- nal resources may be cut or otherwise diminished during budgetary cycles. Reorientation of the program, retraining of staff, and repriori- tization of the work may be required. This is an essential aspect of public management. If the indicator is new, be careful about setting firm targets. It might be preferable to use a range instead. A target does not have to be a single numerical value. In some cases it can be a range. For ex- ample, in 2003, one might set an education target that states "by 2007, 80 to 85 percent of all students who graduate from secondary school will be computer literate." It takes time to observe the effects of improvements, so be realistic when setting targets. Many development and sector policies and pro- grams will take time to come to fruition. For example, environmental reforestation is not something that can be accomplished in one to two years. Finally, it is also important to be aware of the political games that are sometimes played when setting targets. For example, an organiza- tion may set targets so modest or easily achieved that they will surely be met. Another game that is often played in bureaucracies is to move the target as needed to fit the performance goal. Moving tar- gets causes problems because indicator trends can no longer be dis- cerned and measured. In other cases, targets may be chosen because they are not politically sensitive. Examples of Targets Related to Development Issues Box 5.1 presents two examples of targets related to development is- sues. One should work toward setting a specific target by identifying the concerned groups, the objective, and the timeframe by which the target is to be achieved. In each case the target will be just the first of several sequential sets of targets needed to reach the outcome. Fur- thermore, each sequential target is set from the baseline data estab- lished in the previous step. Targets should specify what is being tracked, the expected amount 94 Ten Steps to a Results-Based Monitoring and Evaluation System Box 5.1 Examples of Development Targets 1. Goal: Economic Well-Being Outcome target: By 2008, reduce the proportion of people living in extreme poverty by 20 percent against the baseline. 2. Goal: Social Development Outcome target: By 2008, increase the primary education enrollment rate in the Kyrgyz Republic by 30 percent against the baseline. of change or improvement, and a timeframe by which the target will be achieved. The Overall Performance-Based Framework The completed matrix of outcomes, indicators, baselines, and targets becomes the performance framework. It defines outcomes and plans for the design of a results-based M&E system that will, in turn, begin to provide information on whether interim targets are being achieved on the way to the longer-term outcome. Figure 5.3 illustrates the completed performance framework for a national education development policy area. The traditional imple- mentation dimensions of inputs, activities, and outputs also need tar- gets, as they always have; we are emphasizing here that now out- comes need targets as well. The performance framework becomes the basis for planning-- with attendant implications for budgeting, resource allocation, staffing, and so forth. The framework can and should be a relevant guide to managers. It should be frequently consulted and considered during the process of managing toward the desired outcomes. These performance frameworks have broad applicability, and can be usefully employed as a format for national poverty reduction strategies, as well as framing project, program, and policy outcomes. Performance targeting is critical to the process of reaching out- comes. The formula for arriving at the target performance is a simple one involving baseline indicator levels and desired levels of improve- ment over a specified period of time. A participatory, collaborative process with relevant stakeholders and partners is also key. Planning for Improvement--Selecting Results Targets 95 Figure 5.3 Developing Targets for One Policy Area Example: Education Outcomes Indicators Baselines Targets 1. Nation's children 1. Percent of eligible 1. In 1999, 75 percent 1. By 2006, 85 percent have better access urban children en- of children ages 3­5 of children ages 3­5 to preschool rolled in preschool programs education 2. Percent of eligible rural 2. In 2000, 40 percent 2. By 2006, 60 percent of children enrolled in of children ages 3­5 children ages 3­5 preschool education 2. Primary school 1. Percent of Grade 6 1. In 2002, 75 percent 1. By 2006, 80 percent learning outcomes students scoring 70% scored 70 percent or scoring 70 percent or for children are or better on better in math, and better in math and 67 improved standardized math 61 percent scored percent scoring 70 and science tests 70 percent or better percent or better in in science science Chapter 6 Step 6: Monitoring for Results Figure 6.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 6 6 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization PART 1 After selecting targets and completing the performance-based frame- work, we are now ready to use the information to monitor for results (figure 6.1). This chapter describes putting together a system to get the necessary data to better inform the decisionmaking process. The resulting data will provide evidence on performance and flag any changes that may be needed for a given project, program, or policy. This chapter focuses on how a results-based M&E system is, most importantly, a system to help government (or any organization) bet- ter manage resources. It now becomes relevant to review the need to manage inputs as well as outputs and outcomes. Managers use a va- riety of organizational tools to manage inputs, including budgets, staffing plans, and activity plans. A results-based M&E system needs to align with annual plans and other work plans of the organization to become a true results-oriented system. 96 Monitoring for Results 97 Figure 6.2 Sample Gant Chart Task -- Phase I Duration Start Finish 1. Recruit personnel 14 days 6/1 6/14 2. Assign project roles and 3 days 6/15 6/17 responsibilities 3. Visit sites 14 days 6/18 7/2 4. Analyze data 10 days 7/3 7/12 5. Draft report 10 days 7/13 7/22 June July 6/1 6/14 6/17 6/18 7/2 7/12 7/22 Source: Authors' data 2004. But a results-based system is not the same as monitoring against a set of annual work plans. Monitoring work plans, however, is very much the way a manager traditionally would assess how well a proj- ect, program, or policy is being implemented. In this traditional ap- proach, a manager's first step might be to identify activities and as- sign responsibilities. Often, a manager might employ the use of an activity chart or Gant chart which is, in essence, a to-do list of activi- ties plotted against a specific time line, showing start and due dates for each item, and who will be responsible for which activities. A typical Gant chart is shown in figure 6.2. A Gant chart is a management tool used to track activities and outputs. However, this management tool does not show whether de- sired results are actually being achieved. Completing all activities mapped in such a chart does not mean that the organization is achieving its desired goals or outcomes. Moreover, focusing on activities and outputs does not mean that 98 Ten Steps to a Results-Based Monitoring and Evaluation System individuals within the organization are not working hard. In many cases, individuals are busy and keeping focused day in and day out. But focused on what? A results-based M&E system focuses the or- ganization on achieving outcomes, and manages to each indicator, as we have established in earlier chapters. An activity-based manage- ment system focuses the organization on working against a set of identified activities, without aligning these activities to outcomes, making it difficult to understand how the implementation of these activities results in improved performance. Be careful not to fall into the trap of equating being busy with being effective. Activities are crucial. They are the actions taken to manage and implement programs, use resources, and deliver the services of gov- ernment. But the sum of these activities may or may not mean the outcomes have been achieved. Another difference between a results-based system and an activi- ties-based system is that, with an activities-based work plan, one looks at whether the activities were completed in a timely and appro- priate manner. Monitoring systems, however, demonstrate whether results have been achieved. It is the effective use of resources that counts, not just their efficient use. This chapter considers (a) key types and levels of monitoring; (b) links between implementation monitoring and results monitoring; (c) key principles in building a monitoring system; (d) the needs of every monitoring system; (e) the data quality triangle; (f) analyzing performance data; (g) achieving results through partnership; and (h) pretesting data collection instruments and procedures. Key Types and Levels of Monitoring As figure 6.3 indicates, there are two key types of monitoring--im- plementation monitoring and results monitoring. Both are important in tracking results. Figure 6.4 provides examples of results monitoring at the policy, program, and project levels. Implementation monitoring tracks the means and strategies (that is, those inputs, activities, and outputs found in annual or multiyear work plans) used to achieve a given outcome. These means and strategies are supported by the use of management tools, including budgetary resources, staffing, and activity planning. Monitoring for Results 99 Figure 6.3 Results-Based Monitoring · Long-term, widespread Goal improvement in society (impacts) Results · Intermediate effects of Outcomes outputs on clients · Products and services Outputs produced · Tasks personnel undertake Activities to transform inputs to outputs Implementation · Financial, human, and Inputs material resources Source: Binnendijk 2000. It should also be noted that there is an interaction between means and strategies (inputs, activities, and outputs) and outcome targets. Targets are set according to what the means and strategies potentially can yield. We have spent much of this handbook examining results-based monitoring and evaluation. But implementation--how well outputs are achieved using available inputs and activities--also needs to be measured. Next, the alignment of the outputs with the results the or- ganization hopes to achieve over time needs to be examined. This 100 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 6.4 Examples of Results Monitoring Infant Health Girls Education Policy Decreasing infant Increasing girls monitoring mortality rates educational attainment Program Clinic-based prenatal Number of girls in monitoring care is being used secondary schools by pregnant women completing math and and science courses Project Information on good Number of girls monitoring prenatal care provided in four urban in six targeted villages neighborhoods completing primary education brings us closer to the concept of performance budget frameworks. A performance budget framework is an expenditure planning sys- tem that assumes good macroeconomic and fiscal management, sec- tor priority setting, and program performance management. Budgets are developed according to funds available for a given budget year, with managers stating outputs they will achieve over that budget year. A medium-term budget incorporates the idea that three one- year budgets should be used to achieve desired targets or outcomes. Thus, performance-based budgets budget to outputs, but also help officials manage to outcomes. Boxes 6.1 and 6.2 review results-monitoring efforts in Mexico and Brazil. The lessons we can draw from these various experiences include the following: · If a strong link is to be forged between performance monitoring and resource allocation, a single unit must be responsible for both. · If performance is intended to influence management, a single unit must be responsible for carrying out activities and monitor- ing performance. · The units responsible for performance monitoring, management, and resource allocation must coincide for accountability to be Monitoring for Results 101 Box 6.1 Results Monitoring in Mexico Mexico has separate planning and budget processes. A National Development Plan is prepared every six years, roughly coterminous with the president's term of office. Objectives, policies, and performance targets are set through this process. Programs (the mode for achieving the objectives) derive from the plan. The annual budget process takes the objectives and programs as given. After determining annual resource con- straints, funds are allocated to programs and projects. Performance information is incorporated into the annual budget documents--some 10,000 indicators primarily measuring performance relative to plan tar- gets. But these performance measures are not used in agency management decisions, nor are they used in resource allocation decisions. The Office of the President does monitor these indicators, but follow-up is unclear, and there are no formal reviews. Performance is not built into pay either. Moreover, the program structure has changed annually over the past few years, suggesting it does not tie into an organizational structure or a program manager; therefore the accountability framework is weak. Source: Dorotinsky 2003b. possible, and to enable improvements in efficiency and effective- ness (or even to enable monitoring of efficiency or effectiveness). Links between Implementation Monitoring and Results Monitoring Figure 6.5 depicts how outcomes and targets link to annual work plans, as well as the continuous flow of information up and down the system. Annual work plans are the means and strategies that are used by the organization to use inputs effectively to achieve outputs and, ultimately, outcomes and impacts. We learned in chapter 5 that every target is an interim effort on the way to achieving an outcome. Thus, a means and strategy should be implemented to help achieve every target. The example of children's morbidity in figure 6.6 illustrates the links between means and strategies, target, outcome, and impact, that is, the specific links between implementation monitoring and re- sults monitoring. In this example, one target--reducing the incidence of gastrointestinal disease by 20 percent over three years--has been identified to help reach the outcome of improving children's health. A manager would next identify an annual strategy aimed at reducing the incidence of gastrointestinal disease by the targeted amount. In 102 Ten Steps to a Results-Based Monitoring and Evaluation System Box 6.2 Results Monitoring in Brazil Countries struggle to integrate performance information and management oversight--to have performance information actually used in decisionmaking. Some attempt to monitor actual performance relative to prior baseline performance or benchmarks, while others seek to monitor performance relative to predetermined targets or plans. The approach chosen, and vehicles for implementation, are influenced by the degree to which national planning and budgeting processes are integrated. Brazil has a national plan separate from the budget process. The Ministry of Planning, Budget, and Management is responsible for developing the five-year plan (roughly coterminous with the presidential term of office). The planning process is used to set priorities, objectives, and performance targets. (Unlike in Mexico, the program structure is fixed, and covers all government activities. Also unlike Mexico, the na- tional plan includes resource allocations for programs, by year, over the planning period.) But, given the fixed, multiyear nature of the plan, target resource allocations beyond the first year are highly uncertain. New administrations imprint their policies according to which programs they select as priority programs, with targets and resource allocations designated for the programs. For example, the Cordoso administra- tion designated 80 priority programs. A management information system was developed to tie program funding to performance information, focusing on performance relative to plan targets. Programs were defined supra-organizationally--cutting across ministries and implementing agencies--and program managers were appointed to manage the pro- grams. However, the program managers had no formal authority, controlled no resources, and could not actually influence the activities in ministries that participated in their programs (except in a few cases where the head of a ministry activity was also designated the program manager). Under this structure, the activity manager cannot use program performance information to manage work, and program managers have no influence over actual management. There is a mismatch between authority and responsibility that prevents accountability. In Brazil, performance information is not included in the formal budget documents, but the on-line database does allow partial matching of objectives, performance, and resources--marginal resources. In developing the program concept, Brazil created separate "programs" to encompass personnel expenses, so all other programs only contain the marginal cost of the activity. Despite the structural flaws in the system, Brazil did try to stimulate management use of performance in- formation. The planning office of the Ministry of Planning, Budget, and Management used the information system for quarterly performance updates. The planning office used this information to evaluate each pri- ority program with respect to national plan targets and financial performance relative to a given year's budget. Programs performing poorly, or not likely to fully use that year's resources, would lose resources that would then be transferred to other priority programs deemed to be performing better. This was an at- tempt to use performance information for management and resource decisions, and give added imperative to performance improvement. Source: Dorotinsky 2003b. Monitoring for Results 103 Figure 6.5 Links between Implementation Monitoring and Results Monitoring Outcome Target 1 Target 2 Target 3 Means and Means and Means and strategies (multi- strategies (multi- strategies (multi- year and annual year and annual year and annual work plans) work plans) work plans) doing so, the manager would need to take into account the inputs available over three budget years, and decide how to plan the organ- ization's work to achieve the stated target. PART 2 Key Principles in Building a Monitoring System There are a number of key principles involved in building a results- based monitoring system: · There are results information needs at the project, program, and policy levels. · Results information must move both horizontally and vertically in the organization (sometimes presenting a political challenge). 104 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 6.6 Linking Implementation Monitoring to Results Monitoring Children's mortality reduced Goal Children's morbidity reduced Outcome Reduce incidence of childhood gastro- intestinal disease by 20 percent over Target three years · Improve cholera prevention programs Means and · Provide vitamin A supplements Strategies · Encourage use of oral rehydration therapy · Demand for results information at each level needs to be identified. · Responsibility at each level needs to be clear for 1. What data are collected ( source) 2 When data are collected (frequency) 3. How data are collected (methodology) 4. Who collects data 5. Who reports data 6. For whom data are collected. Performance information needs to move both horizontally and vertically within and between organizations. Horizontal sharing of Monitoring for Results 105 information is crucial. People need to know and understand what in- formation is being collected by their own organization and by other organizations. For instance, there might be one organization that is collecting data that would be suitable for another. In addition, if each organization starts its own information system, there may not be suf- ficient capacity to sustain all of the systems. Many organizations find it difficult to share information horizon- tally. Information may move easily in a vertical manner within a sys- tem, but often there are strong political and organizational walls be- tween one part of the system and another. Bureaucratic and political turf battles are often the cause. Also, bureaucratic incentives are al- most always vertical; seldom are there incentives to share informa- tion horizontally. Ideally, all concerned organizations and agencies need to coordi- nate and collaborate in sharing performance information, especially in those instances where there are intra-institutional partnerships de- veloped to achieve specific targets. It is important to be as clear and precise as possible in the answers to the six questions about responsibility for the system. If these six questions cannot be answered, there will likely be gaps and the sys- tem may falter. This is yet another reason to begin by piloting the ini- tiation of a performance-based M&E system. Achieving Results through Partnership More and more partnerships are being formed to achieve develop- ment goals. Partnerships may be formed at the international and multilateral, regional, country, and governmental levels. Whatever the case, the same results-based monitoring system can be applied to partnership efforts, as illustrated in figure 6.7. Given scarce resources and ambitious development objectives, de- velopment partners need to leverage resources to achieve the desired goal. Therefore, the means and strategies will be set by multiple part- ners. One must look beyond one's own organizational unit when considering available inputs. Partnerships may be created elsewhere in one's own organization or even with other organizations inside or outside the government. When resources are cut or diminished, governments and organiza- tions may need--or be forced to enter into--partnerships with others to reach goals that may be similar. Collaborations can include the for- 106 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 6.7 Achieving Results through Partnership Goal Outcome Outcome Outcome Target 1 Target 2 Means and Strategy Means and Strategy Means and Strategy Partner 2 Partner 2 Partner 2 Partner 1 Partner 3 Partner 1 Partner 3 Partner 1 Partner 3 mation of partnerships with the private sector, NGOs, and the inter- national donor community. By combining resources, outcomes are more achievable--even during times of input constraints. Needs of Every Results-Based Monitoring System Every monitoring system needs four basic elements: ownership, man- agement, maintenance, and credibility (figure 6.8). Ownership Ownership can be thought of as the demand part of the equation. Ownership has to come from those at every level who use the system, and demand for performance information at each level needs to be identified. Stakeholder ownership of data at every level--national, re- gional, and local--is critical. If there are levels where people do not see the need for, or have a use for, the data collected, there will be Monitoring for Results 107 Figure 6.8 Every Monitoring System Needs Ownership Management Maintenance Credibility problems with quality control and ownership. The feedback loop will be disrupted. Without ownership, stakeholders will not be will- ing to invest time and resources in the system. The system will ulti- mately degenerate, and the quality of data will decline. A strong political champion can help to ensure ownership of the system. A champion is needed to stress that good performance data must be generated, shared, and properly reported. Management Who, how, and where the system will be managed is critical to its sustainability. Data collection can also be hampered by overlap of data coming from different agencies; duplication of data in min- istries and the national statistical agency; time lags in receiving data, that is, data that are received too late to have an impact on the deci- sionmaking process; and people not knowing what data are available. Maintenance Maintenance of monitoring systems is essential, to prevent the sys- tems from decaying and collapsing. It is important to know who will collect what kind of information and when, and to ensure that infor- mation is flowing horizontally and vertically in the system. Monitor- ing systems, like other government information systems (such as au- diting or budgeting) must be continually managed. 108 Ten Steps to a Results-Based Monitoring and Evaluation System Management and maintenance of M&E systems require creating the right incentives and providing sufficient financial, human, and technical resources for organizations, managers, and staff to carry out monitoring tasks. Individual and organizational responsibilities should be delineated, and a clear "line of sight" established--mean- ing that staff and organizations should understand their connections to common goals. Clear relationships need to be established between actions and results. Individuals and organizations need to understand how their specific tasks contribute to the big picture. Good maintenance of monitoring systems should also take into ac- count new advances in management and technology. Systems, proce- dures, or technologies may need upgrading and modernizing. Staff and managers should also be provided periodic training to keep their skills current. Unless systems are well managed, they will deteriorate. Monitor- ing systems--like any other systems--require constant rebuilding, re- newal, and strengthening through good management. Credibility Credibility is also essential to any monitoring system. Valid and reli- able data help ensure the credibility of the system. To be credible, monitoring systems need to be able to report all data--both good and bad. If bad news, or information demonstrating failure to meet desired outcomes and targets, is deliberately not reported, the system will not be credible. In some instances, political pressure may be brought to bear on national statistical offices to minimize bad news or not report certain data, for instance, HIV incidence, or infant mortality. If political con- straints are such that no negative news or data can be reported, or the messenger is punished, the monitoring system will be compro- mised. In short, if people think information is politically motivated or tainted, they will not trust it and will not use it. The Data Quality Triangle: Reliability, Validity, and Timeliness A data collection system for all indicators (implementation and results) should possess three key criteria: reliability, validity, and timeliness (fig- ure 6.9). To the extent that any of these criteria are absent, the credibil- ity of the system will diminish. (See also Hatry 1999, p. 223.) Monitoring for Results 109 Figure 6.9 Key Criteria for Collecting Quality Performance Data Reliability Validity Timeliness Reliability is the extent to which the data collection system is stable and consistent across time and space. In other words, measurement of the indicators is conducted the same way every time (figure 6.10). Figure 6.10 The Data Quality Triangle: Reliability The extent to which the data collection approach is stable and consistent across time and space Validity is important: indicators should measure, as directly and succinctly as possible, actual and intended performance levels (figure 6.11). 110 Ten Steps to a Results-Based Monitoring and Evaluation System Figure 6.11 The Data Quality Triangle: Validity The extent to which indicators clearly and directly measure the performance intended to be measured Timeliness consists of three elements: frequency (how often data are collected); currency (how recently data have been collected); and accessibility (data availability to support management decisions) (fig- ure 6.12). If the data are not available to decisionmakers when they need it, the information becomes historical data. Modern public management requires good and timely information. Real-time, con- tinuous data that decisionmakers can use to lead and manage in their work environment is now essential. It makes little sense to manage in the public sector using essentially historical data that may be three, four, or even five years old. Figure 6.12 The Data Quality Triangle: Timeliness · Frequency (how often are data collected?) · Currency (how recently have data been collected?) · Relevance (are data available frequently enough to support management decisions?) Monitoring for Results 111 Analyzing Performance Data Performance findings should be used to help improve projects, pro- In analyzing and reporting grams, and policies. Analyzing and reporting data yields important, data, the more frequent the continuous information about the status of projects, programs, and data measurements over policies. It can also provide clues to problems that arise during the time, the more certain one course of implementation, and create opportunities to consider im- can be of trends, direc- provements in implementation strategies. The continuous stream of tions, and results. data can also provide significant information regarding trends and directions over time. The more often measurements are taken, the less guesswork there will be regarding what happened between specific measurement inter- vals (figure 6.13). More data points enable managers to track trends Figure 6.13 Analyzing Results Data Examine changes over time: · Compare present to past data to look for trends and other changes. · The more data points there are, the more compelling the trends. Access to Access to rural markets rural markets ? Time Time and understand project, program, and policy dynamics. The more time that passes between measurements, the greater the chance that events and changes in the system might happen that may be missed. For example, if there is a year between measurements, many things can hap- pen and it may be more difficult to attribute causality. Did the indicator get better? Worse? Was there a straight-line progression or a wave? Consequently, the monitoring system strategy should include a clear data collection and analysis plan detailing the following: 112 Ten Steps to a Results-Based Monitoring and Evaluation System · Units of analysis (for example, school district, community There is often an explicit hospital, village, region) tradeoff between measure- · Sampling procedures ment frequency and meas- · Data collection instruments to be used urement precision. Cost · Frequency of data collection and capacity also come · Expected methods of data analysis and interpretation into play in making deci- · Those responsible for collecting the data sions about how often and · Data collection partners, if any how precisely to measure · Those responsible for analyzing, interpreting, and reporting data indicators. · For whom the information is needed · Dissemination procedures · Follow-up on findings. Pretesting Data Collection Instruments and Procedures Pretesting or piloting data collection instruments and procedures is vital to building an effective monitoring system. Key points about pretesting include the following: In short, do not move too · A data collection approach needs be be tested to find out how quickly. Start on a small good it is. scale and pilot whenever · Pretesting provides a way to improve instruments or proce- possible. dures--before data collection is fully under way. · Avoiding pretesting probably will result in mistakes. The mistake could cost the organization a lot of time and money, and maybe its valued reputation with the public. · If there is some ambiguity as to how data will be collected and If the monitoring system is what the data will look like, it is best to pilot several strategies, to be a useful management if possible. tool, it needs to be man- For example, the first set of measurements will be the baseline-- ageable. Do not overload and it may not be exactly what should be measured. If the baseline the system with too many is erroneous because the wrong (or incomplete) data are being col- indicators. Otherwise, too lected--and targets have been set against this baseline--the monitor- much time will be spent ing system will be based on a faulty foundation. managing the system that In sum, monitoring for results entails both implementation moni- produces the data, and not toring and results monitoring. It involves the formation of partner- enough time will be spent ships to attain common outcomes. Every monitoring system needs using the data to manage. ownership, management, maintenance, and credibility. Monitoring for results also calls for data collection and analysis of performance data. The key criteria for collecting quality performance data are reli- ability, validity, and timeliness. Finally, pretesting of data collection instruments and procedures is important in every monitoring system. Chapter 7 Step 7: The "E" in M&E--Using Evaluation Information to Support a Results-Based Management System Figure 7.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 6 7 7 8 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization The previous chapters of this handbook placed a strong emphasis on the monitoring function--the "M" in M&E. Building a monitoring system to continuously track performance is absolutely essential for managers. The monitoring system gives ongoing information (via se- lect indicators) on the direction of change, the pace of change, and the magnitude of change. It can also identify unanticipated changes. All are critical to knowing whether policies, programs, and projects are moving in the intended direction. We have also stressed that monitoring data do not give the basis for attribution and causality for change. These monitoring data also do not provide evidence of how changes are coming about--only that they are or are not occurring. Likewise, monitoring data, in and of themselves, cannot address the strengths and weaknesses in the de- sign of the project, program, or policy. Consequently, to address these and other important questions regarding the generation of ap- propriate results, evaluation information is necessary--the "E" in M&E (figure 7.1). 113 114 Ten Steps to a Results-Based Monitoring and Evaluation System We have defined evaluation as an assessment of a planned, ongo- ing, or completed intervention to determine its relevance, efficiency, effectiveness, impact, and sustainability. The intent is to incorporate lessons learned into the decisionmaking process. It is appropriate that we now come to an examination of the eval- uation function in M&E systems. We want to stress the complemen- tarity of evaluation to monitoring. Each supports the other--even as each asks different questions and will likely make different uses of in- formation and analyses. The immediate implication is that moving to a results-based M&E system requires building an information and analysis system with two components--monitoring and evaluation. Either alone, in the end, is not sufficient. There are several complementarities of monitoring and evaluation. First is sequential complementarity, in which monitoring information can generate questions to be subsequently answered by evaluation-- or the reverse, with evaluation information giving rise to new areas or domains of monitoring to be initiated. Second is information com- plementarity, in which both monitoring and evaluation can use the same data, but pose different questions and frame different analyses. Third is interactional complementarity, in which managers are using monitoring and evaluation in tandem to help direct their initiatives. It is important to emphasize here that the evaluation function in the M&E system significantly expands and moves beyond what is understood as the traditional after-the-fact approach to evaluation. Evaluation is not restricted to assessing causes and changes after an intervention or initiative is over. The after-the-fact approach is re- strictive because this type of evaluation information does not feed back into the ongoing management of the government organizations and units aimed at achieving public sector results. The emphasis on after-the-fact evaluations as the means to strive for the definitive an- swers on attribution and causality necessarily precludes real-time uses of evaluation by public sector managers. What follows is not a "how to" on designing and conducting evaluations. There are many textbooks and handbooks that can take a reader through the step-by-step process of an evaluation-- from design, methods selection, data collection and analysis, to re- porting and dissemination. One electronic source for this material and guidance comes in 12 modules from the International Program in Development Evaluation Training (IPDET) and can be found at http://www.worldbank.org/oed/ipdet/ (World Bank 2001a). Using Evaluation Information to Support a Results-Based Management System 115 The emphasis is on how the development of an evaluation capacity in government supports a results-based management approach and the uses managers can make of evaluation information. Good evalua- tive information can provide answers to a broad range of questions relevant to performance and the achievement of outcomes. We will identify a number of these questions as well as the evaluation strate- gies available to answer them. Uses of Evaluation The emphasis on building sources of ongoing evaluation information versus sporadic and individual evaluation studies spaced out over generally lengthy periods is deliberate. M&E systems need to provide government officials with useful and timely information to manage and guide government resources and interventions. The value of an evaluation comes from its use. Pragmatic Uses of Evaluation While the evaluation literature is replete with long and technical dis- cussions of different types and categories of use, this material will be bypassed. Instead, we will go to a pragmatic list of six uses that gov- ernment managers can make of evaluation information. Help Make Resource Allocation Decisions Evaluation information can inform managers on what policies or programs have been more or less successful in terms of their outcomes and thus what level of resources they might merit. Likewise, evaluation informa- tion can help guide decisions on whether the results of pilot efforts suggest expanding, redesigning, or even dropping the initiative alto- gether. Help Rethink the Causes of a Problem Frequently, policy and pro- gram interventions appear not to be having any notable conse- quences on an existing problem. While the absence of change may be attributable to either poor design or poor implementation, it may also be that the intervention is of no consequence because the prob- lem is different than originally presumed. Evaluation information can raise the need for a re-examination of the presumed cause of a problem--and what alternative countermeasures might be needed. 116 Ten Steps to a Results-Based Monitoring and Evaluation System Identify Emerging Problems Evaluation information can highlight issues that are not yet widespread, but may clearly require the atten- tion of government officials, such as rising drop out rates in select groups of youth, the number of orphans whose parents have died from AIDS, or drug use among subteens. Support Decisionmaking on Competing or Best Alternatives Often governments will approach a problem situation by piloting more than one strategy. For example, a government may try to address youth unemployment through in-school programs, special apprentice programs in the private sector, vouchers for employers who hire youth, and so forth. After each pilot has been in operation for some time, it will be easier to determine which has the more compelling evidence of success, and which merits more or less support. Support Public Sector Reform and Innovation Evaluation informa- tion can provide evidence to citizens that reform efforts are working. For example, evidence that school improvements are being made, that corruption is being diminished, or that more of the rural poor are receiving health care can give credibility to government efforts. Reform efforts often lose momentum if there is no evidence of posi- tive change. Build Consensus on the Causes of a Problem and How to Respond Evaluation information can contribute to the discussions among government officials and important stakeholders about the causes of the conditions and how to create an appropriate response. The defi- nition of a problem should precede any deployment of countermea- sures to try and solve, or at least diminish, the problem. Evaluation information can provide evidence of causality, and evidence of the relevance and impact of previous responses. To summarize this brief examination of the uses of evaluation in- formation in an M&E system, government officials and their part- ners can use this information to focus on the broad political strategy and design issues ("are we doing the right things?"), on operational and implementation issues ("are we doing things right?"), and whether there are better ways of approaching the problem ("what are we learning?"). See box 7.1. Using Evaluation to Answer Management Questions Evaluations can also help answer eight different types of questions that managers frequently pose: Using Evaluation Information to Support a Results-Based Management System 117 Box 7.1 Evaluation Provides Information on: · Strategy: are the right things being done? -- Rationale or justification -- Clear theory of change · Operations: are things being done right? -- Effectiveness in achieving expected outcomes -- Efficiency in optimizing resources -- Client satisfaction · Learning: are there better ways? -- Alternatives -- Best practices -- Lessons learned · Descriptive: Describe the content of the information campaign in country X for HIV/AIDS prevention. (Focuses on careful descrip- tion of a situation, process, or event. Often used as the basis for a case study approach.) · Normative or compliance: How many days during the year were national drinking water standards met? (Determines whether a project, program, or policy met stated criteria.) · Correlational: What is the relation between the literacy rate and number of trained teachers in a locality? (Shows the link between two situations, or conditions, but does not specify causality.) · Impact or cause and effect: Has the introduction of a new hybrid seed caused increased crop yield? (Establishes a causal relation between two situations or conditions.) · Program logic: Is the sequence of planned activities likely to in- crease the number of years girls stay in school? (Assesses whether the design has correct causal sequence.) · Implementation or process: Was a project, program, or policy to improve the quality of water supplies in an urban area imple- mented as intended? (Addresses whether implementation oc- curred as planned.) · Performance: Are the planned outcomes and impacts from a pol- icy being achieved? (Establishes links between inputs, activities, outputs, outcomes, and impacts.) 118 Ten Steps to a Results-Based Monitoring and Evaluation System · Appropriate use of policy tools: Has the government made use of the right policy tool in providing subsidies to indigenous farmers to deploy a new agricultural technology? (Establishes whether the appropriate instruments were selected to achieve aims.) The Timing of Evaluations Evaluation information is relevant and helpful to government man- agers at all phases of management of policies, programs, and proj- ects. The question of timing is easily answered: Any time there are concerns for which evaluation information can be useful is the time to gather evaluative information. But it is necessary to go deeper in addressing when to deploy re- sources to gather evaluation information. Four instances follow that warrant evaluation information to support management decision- making. (We recognize there are others beyond these four, but these are illustrative of when we think evaluation information is essential.) Divergence between Planned and Actual Performance When regular measurements of key indicators suggest a sharp diver- gence between planned performance and actual performance, evalua- tion information can be crucial. Consider the graphs in figure 7.2. In the graphs in figure 7.2 it is apparent that planned and actual performances are diverging. The manager needs to know why. "What is going on that either we are falling behind our planned per- formance so badly (left chart) or that we are doing so well that we are ahead of our own planning frame (right chart)?" Managers will recognize from their own experience that planned and actual per- formances are most often not identical, and some variation is to be Figure 7.2 Using Evaluation to Explain Performance Divergence Planned Actual Using Evaluation Information to Support a Results-Based Management System 119 expected. But when that divergence is dramatic, sustained, and has real consequences for the policy, program, or project, it is time to step back, evaluate the reasons for the divergence, and assess whether new strategies are needed (in the case of poor performance), or learn how to take the accelerated good performance and expand its appli- cations. The Contributions of Design and Implementation to Outcomes Evaluation information can help differentiate between the contribu- tions of design and implementation to outcomes. In figure 7.3, Square 1 is the best place to be--the design (a causal model of how to bring about desired change in an existing problem) is strong and the implementation of actions to address the problem is also strong. All managers, planners, and implementers would like to spend their time and efforts like this--making good things happen for which there is demonstrable evidence of positive change. Square 2 generates considerable ambiguity in terms of perform- ance on outcome indicators. In this situation there is a weak design that is strongly implemented--but with little to no evident results. The evidence suggests successful implementation, but few results. The eval- Figure 7.3 Using Evaluation to Determine the Impacts of Design and Implementation on Outcome Strength of design High Low High 1. 2. Strength of implementation Low 3. 4. 120 Ten Steps to a Results-Based Monitoring and Evaluation System uative questions would turn to the strength and logic of the design. For example, was the causal model appropriate? Was it sufficiently robust that, if implemented well, it would bring about the desired change? Was the problem well understood and clearly defined? Did the pro- posed change strategy directly target the causes of the problem? Square 3 also generates considerable ambiguity in terms of per- formance with respect to outcome indicators. In this situation there is a well-crafted design that is poorly implemented--again, with little to no evident results. This is the reverse situation of Square 2, but with the same essential outcome--no clear results. The evaluative ques- tions focus on the implementation processes and procedures: Did what was suppose to take place actually take place? When, and in what sequence? With what level of support? With what expertise among the staff? The emphasis is on trying to learn what happened during implementation that brought down and rendered ineffective a potentially successful policy, program, or project. Square 4 is not a good place to be. A weak design that is badly im- plemented leaves only the debris of good intentions. There will be no evidence of outcomes. The evaluation information can document both the weak design and the poor implementation. The challenge for the manager is to figure out how to close down this effort quickly so as to not prolong its ineffectiveness and negative consequences for all involved. Resource Allocations When resource allocations are being made across policies, programs, or projects, evaluation information can help managers analyze what is or is not working efficiently and effectively. The tradeoffs in bud- get and personnel allocations are many. Political conflicts among competing demands are real. Evaluation information can assist in the process, especially when the government is working to install a per- formance-based budget system. But it is also important and realistic to acknowledge that evaluation information cannot override and negate political, institutional, or personal agendas that inevitably come into play. Conflicting Evidence of Outcomes Evaluation information can help when similar projects, programs, or policies are reporting different outcomes. Comparable initiatives with clearly divergent outcomes raise the question of what is going on and where. Among the questions that evaluation information can address Using Evaluation Information to Support a Results-Based Management System 121 are the following: Are there strong variations in implementation that are leading to the divergence? Or do key individuals not understand the intentions and rationale of the effort, so are providing different guidance leading to essentially different approaches? Or, as a third possibility, are the reporting measures so different that the compar- isons are invalid? Types of Evaluations Different types of evaluations are appropriate for answering different kinds of questions. There is no "one size fits all" evaluation template to put against the variety of questions. It is important for managers to have an understanding of what they want to know from evalua- tions. Likewise, it is important for those producing the evaluative in- formation to understand what is needed by the manager. It is not beneficial for anyone involved to find themselves with a mismatch between the question asked and the information provided. Figure 7.4 depicts seven broad evaluation strategies that can be used to generate evaluation information. Each is appropriate to spe- cific kinds of evaluation questions, and each will be briefly reviewed. (Note that only one of these seven is the classic after-the-fact evalua- tion--the impact evaluation.) Figure 7.4 Seven Types of Evaluations Performance Pre-implementation logic chain assessment assessment Process implementation Rapid appraisal Case study evaluation Impact evaluation Meta-evaluation 122 Ten Steps to a Results-Based Monitoring and Evaluation System Performance Logic Chain Assessment The performance logic chain assessment evaluation strategy is used to determine the strength and logic of the causal model behind the policy, program, or project. The causal model addresses the deploy- ment and sequencing of the activities, resources, or policy initiatives that can be used to bring about a desired change in an existing condi- tion. The evaluation would address the plausibility of achieving that desired change, based on similar prior efforts and on the research lit- erature. The intention is to avoid failure from a weak design that would have little or no chance of success in achieving the intended outcomes. In attempting to assess the present effort in comparison to past ef- forts, the evaluator could focus on the level of resources, timing, ca- pacity of the individuals and organizations involved, level of ex- pected outcomes, and so forth, to determine if the present strategy can be supported from prior experience. Likewise, in examining the research literature, the evaluator can find out if the underlying prem- ises of the proposed initiative can be supported; for example, that in- creased awareness by citizens of government corruption through a public information campaign will lead to increased pressure from civil society for the government to combat and control the corruption. Pre-Implementation Assessment The pre-implementation assessment evaluation strategy addresses three standards that should be clearly articulated before managers move to the implementation phase. The standards are encompassed in the following questions: Are the objectives well defined so that outcomes can be stated in measurable terms? Is there a coherent and credible implementation plan that provides clear evidence of how im- plementation is to proceed and how successful implementation can be distinguished from poor implementation? Is the rationale for the deployment of resources clear and commensurate with the require- ments for achieving the stated outcomes? The intention of such an evaluation approach is to ensure that failure is not programmed in from the beginning of implementation. Process Implementation Evaluation The focus of process implementation evaluation is on implementa- tion details. What did or did not get implemented that was planned? What congruence was there between what was intended to be imple- Using Evaluation Information to Support a Results-Based Management System 123 mented and what actually happened? How appropriate and close to plan were the costs; the time requirements; the staff capacity and capability; the availability of required financial resources, facilities, and staff; and political support? What unanticipated (and thus unintended) outputs or outcomes emerged from the implementation phase? The implementation phase can be short or long. The emphasis throughout would be to study the implementation process. Man- agers can use this information to determine whether they will need to make any mid-course corrections to drive toward their stated out- comes. This evaluation strategy is similar to monitoring. The added value is that the implementation is not just documented (monitored). In evaluating the implementation, unanticipated outcomes can be stud- ied. Additionally, some of the more intangible aspects of implementa- tion, such as political support, institutional readiness for change, and the trust in management to successfully lead a change effort, can be addressed. Finally, having some understanding of why the implemen- tation effort is or is not on track gives a firm basis for initiating countermeasures, if needed. Rapid Appraisal Because we view M&E as a continuous management tool, rapid ap- praisals deserve special consideration here. Rapid appraisals can be invaluable to development practitioners in a results-based M&E sys- tem. They allow for quick, real-time assessment and reporting, pro- viding decisionmakers with immediate feedback on the progress of a given project, program, or policy. Rapid appraisal can be characterized as a multimethod evaluation approach that uses a number of data collection methods. These methods tend to cluster in the middle of the continuum presented in figure 4.3. "Rapid appraisal methodology . . . [can be thought of] in the context of the goal of applied research; that is, to provide timely, relevant information to decision-makers on pressing issues they face in the project and program setting. The aim of applied research is . . . to facilitate a more rational decision-making process in real-life cir- cumstances" (Kumar 1993, p. 9). There are five major rapid appraisal data collection methods: (a) key informant interviews; (b) focus group interviews; (c) community inter- views; (d) structured direct observation; and (e) surveys. These meth- ods are particularly useful in dealing with the following situations: 124 Ten Steps to a Results-Based Monitoring and Evaluation System · When descriptive information is sufficient for decisionmaking · When an understanding is required of the motivations and atti- tudes that may affect people's behavior, in particular the behav- ior of target populations or stakeholders in an intervention · When available quantitative data must be interpreted · When the primary purpose of the study is to generate suggestions and recommendations · When the need is to develop questions, hypotheses, and proposi- tions for more elaborate, comprehensive formal studies (Kumar 1993, pp. 21­22). Rapid appraisals are highly relevant to the timely production of man- agement-focused evaluation information. As with any evaluation method, there are some strengths and weaknesses of rapid appraisals that should be taken into account. Rapid appraisals produce needed information on a quick and timely basis and are relatively low cost, especially in comparison with more formal, structured evaluation methods. Such appraisals can provide a quick turnaround to see whether projects, programs, and policies are basically on track. However, the reliability, credibility, and validity of rapid appraisals may be more open to question because of such fac- tors as individual bias and preconceptions, and lack of quantitative data that can be easily replicated and verified. Likewise, it is difficult to aggregate the findings from multiple rapid appraisals, as each is relatively unique and the mix of methods varies from one application to another. On balance, though, rapid appraisals can make rapid re- porting possible and help flag the need for continuous corrections. Case Study The case study is the appropriate evaluation strategy to use when a manager needs in-depth information to understand more clearly what happened with a policy, program, or project. Case studies imply a tradeoff between breadth and depth in favor of the latter. There are six broad ways that managers can draw on case study in- formation to inform themselves: (a) case studies can illustrate a more general condition; (b) they can be exploratory when little is known about an area or problem; (c) they can focus on critical instances (high success or terrible failure of a program); (d) they can examine select instances of implementation in depth; (e) they can look at pro- gram effects that emerge from an initiative; and, finally, (f) they can provide for broader understanding of a condition when, over time, Using Evaluation Information to Support a Results-Based Management System 125 the results of multiple case studies are summarized and a cumulative understanding emerges. Impact Evaluation An impact evaluation is the classic evaluation (though not only after the fact) that attempts to find out the changes that occurred, and to what they can be attributed. The evaluation tries to determine what portion of the documented impacts the intervention caused, and what might have come from other events or conditions. The aim is attribution of documented change. This type of evaluation is diffi- cult, especially as it comes after the end of the intervention (so that if outcomes are to be evident, they will have had time to emerge). Ob- viously, the longer the time between the intervention and the attempt to attribute change, the more likely it is that other factors will inter- fere in either positive or negative ways to change the intended out- come, that the timeframe in which one was seeking to measure change is incorrect (too soon or too late), and that the outcome will become enveloped in other emerging conditions and be lost. Another way of addressing the issue of attribution is to ask the counterfactual question, that is, what would have happened if the intervention had not taken place? Answering this question is difficult. But there are strategies for doing so, using both experimental and quasi-experimental designs. Use of random assignment and control or comparison groups are the basic means of addressing this question. When possible, it is best to plan for impact evaluations before the intervention even begins. Determining which units will receive the in- tervention and which will not, and establishing baseline information on all units, are just two of the reasons for planning the impact eval- uation prospectively. Meta-Evaluation If a number of evaluations have been conducted on one or similar initiatives, a meta-evaluation establishes the criteria and procedures for systematically looking across those existing evaluations to sum- marize trends and to generate confidence (or caution) in the cross- study findings. Meta-evaluation can be a reasonably quick way of learning "what do we know at present on this issue and what is the level of confidence with which we know it?" Leeuw and Cooksy (2003) used a meta-evaluation approach to summarize findings from 126 Ten Steps to a Results-Based Monitoring and Evaluation System three evaluations from three development agencies--the Department for International Development (DIFD), the UNDP, and the World Bank. Characteristics of Quality Evaluations If managers are going to rely on information from an M&E system, they are right to question the quality and trustworthiness of the in- formation they are getting. Poor, inaccurate, and biased information is of no use to anyone. How is a manager to know if the information is worth consider- ing? Without going into a detailed discussion of the many facets of data validity and reliability, and without expecting the manager to have mastered advanced statistics, there are six characteristics that can be considered (figure 7.5). An assessment across these six charac- teristics will not guarantee that the information is impeccable or that it is error free, but it will provide a checklist for a manager to use in forming an opinion on whether to use the information. · Impartiality: The evaluation information should be free of politi- cal or other bias and deliberate distortions. The information should be presented with a description of its strengths and weak- nesses. All relevant information should be presented, not just Figure 7.5 Characteristics of Quality Evaluations Impartiality Usefulness Stakeholder Technical adequacy involvement Feedback and Value for money dissemination Using Evaluation Information to Support a Results-Based Management System 127 that which reinforces the views of the manager. · Usefulness: Evaluation information needs to be relevant, timely, and written in an understandable form. It also needs to address the questions asked, and be presented in a form desired and best understood by the manager. · Technical adequacy: The information needs to meet relevant technical standards--appropriate design, correct sampling proce- dures, accurate wording of questionnaires and interview guides, appropriate statistical or content analysis, and adequate support for conclusions and recommendations, to name but a few. · Stakeholder involvement: There should be adequate assurances that the relevant stakeholders have been consulted and involved in the evaluation effort. If the stakeholders are to trust the infor- mation, take ownership of the findings, and agree to incorporate what has been learned into ongoing and new policies, programs, and projects, they have to be included in the political process as active partners. Creating a façade of involvement, or denying in- volvement to stakeholders, are sure ways of generating hostility and resentment toward the evaluation--and even toward the manager who asked for the evaluation in the first place. · Feedback and dissemination: Sharing information in an appro- priate, targeted, and timely fashion is a frequent distinguishing characteristic of evaluation utilization. There will be communica- tion breakdowns, a loss of trust, and either indifference or suspi- cion about the findings themselves if: (a) evaluation information is not appropriately shared and provided to those for whom it is relevant; (b) the evaluator does not plan to systematically dis- seminate the information and instead presumes that the work is done when the report or information is provided; and (c) no ef- fort is made to target the information appropriately to the audi- ences for whom it is intended. · Value for money: Spend what is needed to gain the information desired, but no more. Gathering expensive data that will not be used is not appropriate--nor is using expensive strategies for data collection when less expensive means are available. The cost of the evaluation needs to be proportional to the overall cost of the initiative. The emphasis in this chapter has been on the role that evaluation 128 Ten Steps to a Results-Based Monitoring and Evaluation System can and should play in the development of a results-based M&E sys- tem. Evaluation information can be relevant at all phases of a policy, program, or project cycle. Evaluation information can be useful to the needs of the public sector manager if it comes in a timely fashion, is appropriately presented, is technically adequate, addresses ques- tions directly, and is trustworthy. Evaluation and monitoring are complementary and both are needed in a results-based management system. Figure 7.6 Examples of Evaluation Privatizing Resettlement water systems Policy Comparing model Comparing strategies evaluations approaches to used for resettlement privatizing public of rural villagers to water supplies new areas Program Assessing fiscal Assessing the degree to evaluations management of which resettled village government systems farmers maintain previous livelihood Project Assessing the Assessing the farming evaluations improvement in water practices of resettled fee collection rates in farmers in one province two provinces Examples of Evaluation at the Policy, Program, and Project Levels Evaluation information can inform policymakers and program and project managers if their interventions are leading to desired results, and provide important clues as to why implementation strategies are or are not on track. Figure 7.6 presents the kind of information eval- uation can provide for projects, programs or polices in two ex- amples: water privatization systems and resettlement strategies. Chapter 8 Step 8: Reporting the Findings Figure 8.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 6 7 88 9 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization Performance information is to be used as a management tool. Thus, ". . . [R]eporting is too performance information is derived from both monitoring and evalu- often the step to which ation. Both can provide critical, continuous, and real-time feedback evaluators give the least on the progress of a given project, program, or policy. thought." Analyzing and reporting performance findings is a critical step be- cause it determines what is reported, when it is reported, and to (Worthen, Sanders, and whom it is reported. This step also has to address the current techni- Fitzpatrick 1997, p. 407) cal capacity of the organization because it focuses on the method- ological dimensions of accumulating, assessing, and preparing analyses and reports. This chapter focuses specifically on reporting findings and address- ing the following issues: (a) uses of monitoring and evaluation find- ings; (b) knowing the audiences and targeting the appropriate infor- mation to those audiences; (c) presentation of performance data in clear and understandable form; and (d) what happens if performance news is bad. 129 130 Ten Steps to a Results-Based Monitoring and Evaluation System The Uses of Monitoring and Evaluation Findings Monitoring and evaluation reports can play many different roles, and the information produced can be put to very different uses: · To demonstrate accountability--delivering on political promises made to citizenry and other stakeholders · To convince--using evidence from findings · To educate--reporting findings to help organizational learning · To explore and investigate--seeing what works, what does not, and why · To document--recording and creating an institutional memory · To involve--engaging stakeholders through a participatory process · To gain support--demonstrating results to help gain support among stakeholders · To promote understanding--reporting results to enhance under- standing of projects, programs, and policies. Evaluation reports serve many purposes. The central purpose, however, is to "deliver the message"--inform the appropriate audi- ences about the findings and conclusions resulting from the collec- tion, analysis, and interpretation of evaluation information. (Adapted from Worthen, Sanders, and Fitzpatrick 1997.) Know and Target the Audience Know your audiences and how they want to see the information ex- "Some call this `speaking pressed. The interests, expectations, and preferred communications truth to power,' but what medium of the audience should be taken into account. A communica- good is speaking truth if tions strategy should be developed that will address the following power isn't listening? Un- questions: less we find more effective ways to help our audiences · Who will receive what information? listen, all our good works · In what format? are likely to go for naught. · When? How we report our results · Who will prepare the information? is often the difference be- · Who will deliver the information? tween creating a tiny ripple During the ongoing process of determining monitoring and evalua- or making a proper tion findings, it is important to ensure that everyone is informed of splash." progress, and that there are no surprises. If the information system is to provide continuous performance feedback as a management tool, (Wholey, Hatry, and Newcomer 1994, p. 549) continuous communication is also important to the process. Monitor- ing and evaluation results should be continuously disseminated to pro- vide feedback to decisionmakers. Informal (phone, e-mail, fax, conver- Reporting the Findings 131 sations) and formal (briefings, presentations, written reports) commu- nications should be a part of the overall communications strategy. Data should be presented in a short and crisp manner and be rele- vant to the target audience. Only the most important data should be presented. "A . . . report [on findings] obviously cannot be well tar- geted without clear definition of its audience(s) and the types of questions that audience is likely to raise about findings" (Worthen, Sanders, and Fitzpatrick 1997, p. 409). If there are multiple audiences--those involved at the project, pro- gram, and policy levels--the data may have to be packaged and for- matted differently according to the main interests and preferences of each audience. The communications strategy should take into ac- count the challenges in communicating results to different stakehold- ers. Furthermore, "[c]lear the report with all key parties before it is formally presented. This will help to eliminate errors and will also ensure that many points are clarified informally without the embar- rassment of confrontations [later on] . . . " (Valadez and Bamberger 1994, p. 437). One can anticipate that there may be multiple uses of the per- formance findings. Think of this as concentric circles, that is, the target audience forms the inner circle, but there may be uses for the findings beyond the inner circle including those less directly con- cerned or affected. "Evaluators often limit the use of evaluation data to the questions . . . under investigation. The information collected may, and usually does, have meaning and use to others in the organization for purposes well beyond the intent of the original eval- uation study" (Wholey, Hatry, and Newcomer 1994, p. 578). Conse- quently, one should also anticipate further dissemination of perform- ance findings to a broader audience. Typically, the higher up the chain of command, the less need there is for extensive detail and explanation; aggregated, succinct data rel- evant to the specific issue will be more appropriate. For this reason, personal briefings--especially to high-level officials--can be another effective means of communicating performance findings. Further down the managerial chain, it is more likely that more operational data will be desired. Large "data dumps" of information are counterproductive. Know what the decisionmakers want and provide them with the necessary information in the format with which they are most comfortable. This may require tailoring information into the preferred format for each of the decisionmakers and end users. 132 Ten Steps to a Results-Based Monitoring and Evaluation System Decisionmakers may be looking for some indications of action re- quired in response to data findings. They will also be interested in available options (including costs, pros and cons, and the like) with respect to acting on performance findings throughout the monitoring and evaluation process. Furthermore, it is important to highlight the implications of rec- ommended actions throughout the monitoring and evaluation process. "Simply recommending that certain actions be taken is rarely sufficient. Officials will usually want a fuller understanding of the implications of their action. Wise evaluators anticipate this need and provide, whenever possible, best estimates (or perhaps a range of estimates) of both the costs and consequences of the recommenda- tions" (Wholey, Hatry, and Newcomer 1994, p. 563). Continuous re- porting on findings can and should also extend to guiding decision- makers through implementation of recommendations. In terms of follow-up and feedback, one could set up a political process to bring stakeholders and evaluators together to discuss find- ings, insights, alternative actions, and next steps. It would also be useful to " . . . obtain feedback periodically from major constituen- cies, such as elected officials, funders, and the public . . . regarding the usefulness and readability of performance reports. Use the feed- back to help tailor future performance reports to the particular audi- ence" (Hatry 1999, p. 154). Comparisons of performance data over time are critical. Providing Report performance data in data for a specific quarter or year by itself is not useful. To distin- comparison to earlier data guish trends, one needs to begin with baselines. Always report against and to the baseline. the baseline and intermediate measurements to determine whether progress has been sustained, whether there was only a short spurt of improvement, or whether early improvements have all disappeared. Comparing actual outcomes to targets is central to reporting results. Table 8.1 illustrates indicator baselines, current and target measure- ments, as well as percentage differences relative to expected outcomes. Presentation of Performance Data in Clear and Understandable Form It is important to report results data in comparison to earlier data and to the baseline. Comparisons over time are critical. The following data can be reported: · Expenditure or income--cost of, or return on, project, program or policy Reporting the Findings 133 Table 8.1 Outcomes Reporting Format: Actual Outcomes versus Targets Difference Baseline Current Target (percentage Outcome indicator (percent) (percent) (percent) points) Rates of hepatitis (N = 6,000) 30 25 20 ­5 Percentage of children with improved overall overall health status (N = 9,000) 20 20 24 ­4 Percentage of children who show four out of five positive scores on physical exams (N = 3,500) 50 65 65 0 Percentage of children with improved nutritional status (N = 14,000) 80 85 83 +2 Source: Sample data 2004. · Raw numbers--early indications, rough projections, estimates, and so forth · Percentages (for example, percentage of citizens served by a project) · Statistical tests · Organizational units · Geographical locations · Demographics · Client satisfaction scales--high, medium, low. Data should be presented in a simple, clear, and easily understand- able format. Only the most important data should be presented. Acronyms and jargon should be avoided. A minimum of background information should be provided to establish the context. Major points should be stated up front. Findings and recommendations should be organized around key outcomes and their indicators. A separate appendix or report can be used to convey detailed data. There are four dimensions of reporting: written summaries, execu- tive summaries, oral presentations, and visual presentations. Written Summaries To be a useful management tool, the written summary should con- tain an introduction (including purpose of report, evaluation ques- tions, program background, and program goals and objectives). The 134 Ten Steps to a Results-Based Monitoring and Evaluation System summary should contain a description of the evaluation (including evaluation focus, methodology, limitations of methodology, who per- formed the evaluation, and when the evaluation was performed). The report should present data on findings selectively and in an under- standable manner; organize data around study questions, major themes or program components; and use charts and tables. Conclusions should be clearly connected to evidence on perfor- mance. Evidence should be presented to support recommendations. When planning the time needed to prepare the analysis and report- ing format, leave plenty of time to revise. Having a knowledgeable outside reader review the findings and draft report can also be helpful. Executive Summaries Executive summaries should be short (one to four pages). Major find- ings and recommendations should be presented in bullet format. The summary can refer readers to the report or appendices for more de- tails. The executive summary should contain a brief overview, includ- ing the background and purpose of the study. It should also include a brief description of major questions, issues, and research methods. Oral Presentations Oral presentations also can be used, either alone or in conjunction with a written report. In addition to rehearsing and getting feedback, one needs to consider the following in preparing for an oral presentation: · Who is the audience? · What should they remember from the presentation? · How much time is there for the presentation? · What are the available delivery resources? · What handouts should be provided, if any? Oral presentations--like written ones--should be simple, clear, and tailored to the audience. Complex language and detailed data should be avoided. Organization is also important: "Tell them what you will tell them; tell them; tell them what you told them." If possible, use an interactive format with the audience, and be prepared for questions. Visual Presentations Visual presentations--charts, graphs, and maps--are also helpful in highlighting key points and performance findings. They can illustrate directions and trends at a glance. There are a variety of charts (pie, Reporting the Findings 135 flow, column, time series, scatter plot, bar, range, and so forth) and graphs (line, scatter, bar, pie, surface, pictograph, contour, histogram, area, circle, column) that should be considered in presenting data to the target audience. The purpose of charts and tables is to describe, explore, tabulate, "Visual presentations of ev- and compare. Charts and tables can provide impact and visual inter- idence should be governed est, encourage audience acceptance and memory retention, and show by principles of reasoning the big picture. Charts and tables should present data simply and accu- about quantitative evi- rately, and make the data coherent. They should engage the audience. dence. For information dis- Tables are best used for presenting data, and highlighting changes, plays, design reasoning comparisons, and relationships. Charts are better for presenting the must correspond to scien- message. They are useful in depicting organizational structures, tific reasoning. Clear and demonstrating flows, presenting data as symbols, conveying concepts precise seeing becomes as and ideas, and presenting numerical data in visual form. one with clear and precise Effectively designed tables will have the following characteristics: thinking." · Simplicity and accuracy (Tufte 2002, p. 53) · Clearly labeled rows and columns with no abbreviations · Percentages rounded to the nearest whole number · Total numbers · Source of the data. Table 8.2 is an example of an effective table that could be used to demonstrate and report descriptive data. Characteristics of effectively designed charts include the following: · Easily read and appropriate for the delivery, using both upper and lower case (not all caps) and only a few type faces · No busy patterns Table 8.2 Sample Table for Reporting Descriptive Data Gender Differences in Voting Voted in last election Yes No Men (N = 1,000) 75% 25% Women (N =700) 55% 45% Source: Sample data 2004. 136 Ten Steps to a Results-Based Monitoring and Evaluation System · Effective use of white space · Simple · Honest scales · Message conveyed in title · Sufficient data provided with chart so that message is clear · Source of the data · Supporting data in an appendix. Effective charts enable policymakers and decisionmakers to quickly see the current status of a given project, program, or policy-- including trends, directions, delays, problems, successes, and prospects. Charts should be used to provide informative and useful visual aids for continuous reporting of findings. Whether in chart or table form, portraying information graphi- cally is an important part of reporting. Figure 8.2 provides some guidance for the use of graphics. Figure 8.2 contains examples of chart options for the continuous process of reporting findings. There are many different reporting formats including written re- ports and displays. It is important to check with users and stakehold- ers for any preferences for data presentation. Be cautious not to use inappropriate graphs just because they may be popular. What Happens If the M&E System Produces Bad Performance News? One cannot manage by receiving only good news. A good perform- "The value of information ance measurement system is intended to surface problems--not just [often] decreases rapidly bring good news. This is another of the political aspects of results- over time, so essential find- based M&E systems. Reporting on bad news is a critical aspect of ings should be communi- how one distinguishes success from failure. If the difference cannot cated as quickly as possible." be determined, it is likely that both failure and success are being re- (Valadez and Bamberger 1994, warded by managers. A good performance system can serve as a kind p. 437) of early warning system. Performance reports should include explanations (if possible) about poor outcomes and identify steps taken or planned to correct problems (Hatry 1999). Messengers should not be punished for de- livering bad news. Instilling fear of bringing forth bad news will not encourage reporting and use of findings. Figure 8.2 Principles of Graphical Excellence Edward Tufte teaches courses in statistical evidence and information de- sign at Yale University. He is considered one of the major authorities on presenting information in a clear and accurate manner. Here are a few guidelines from his writing. "Graphical excellence is the well-designed presentation of interesting data--a matter of substance, of statistics, and of design." "Graphical excellence consists of complex ideas communicated with clarity, precision, and efficiency." "Graphical excellence is that which gives to the viewer the greatest num- ber of ideas in the shortest time with the least ink in the smallest space." "Graphical excellence is nearly always multivariate." "And graphical excellence requires telling the truth about the data." Source: Tufte 2001, p. 51. Sample Charts for Displaying Information Line graph: trends over time Pie chart: parts of a whole Bar chart: percent distribution Cluster bar chart: comparing several items Combination chart Beware of too much of a good thing Chapter 9 Step 9: Using the Findings Figure 9.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 6 7 8 99 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization After examining effective ways of reporting in the previous chapter, Using results-based find- we turn now to the use of findings emanating from the results-based ings will help inform the monitoring and evaluation system (figure 9.1). We will consider (a) decisionmaking process. the uses of performance findings; (b) additional benefits of using the findings--feedback, knowledge, and learning; and (c) strategies for sharing information. Uses of Performance Findings Using findings to improve performance is the main purpose of build- ing a results-based M&E system. The main point of the M&E system is not simply to generate continuous results-based information, but to get that information to the appropriate users in a timely fashion so that the performance feedback can be used to better manage organiza- tions and governments. Findings can be used in a variety of concrete ways, as shown in box 9.1. 138 Using the Findings 139 Box 9.1 Ten Uses of Results Findings 1. Respond to elected officials' and the public's demands for accountability 2. Help formulate and justify budget requests 3. Help make operational resource allocation decisions 4. Trigger in-depth examinations of what performance problems exist and what corrections are needed 5. Help motivate personnel to continue making program improvements 6. Formulate and monitor the performance of contractors and grantees 7. Provide data for special, in-depth program evaluations 8. Help provide services more efficiently 9. Support strategic and other long-term planning efforts (by providing baseline information and later tracking progress) 10. Communicate better with the public to build public trust. Source: Hatry 1999. With respect to helping formulate and justify budget requests, per- formance information can inform decisions that can lead to budget- ary increases--or reductions. Projects, programs, and policies may be enhanced or expanded based on performance feedback; likewise, they may be cut or eliminated altogether. Managers also have the op- tion of offering incentives (monetary and nonmonetary) to personnel for good performance or sanctions (such as poor employee or man- ager performance reviews) for performance that fails to meet expec- tations or falls short of intended outcomes. In terms of motivating personnel, when civil servants are brought in as partners to the business of government, we see better implemen- tation. Employees throughout the system begin to understand and become more enthusiastic about their contributions toward achieve- ment of the desired goal when they have a "line of sight" between their own actions and the goal. In some OECD countries (Australia and France, for example), managers are given greater operational flexibility in exchange for enhanced accountability. Australia provides an example regarding the performance of con- Bringing stakeholders into tractors and grantees. In Australia, there are actual performance con- cooperation with govern- tracts with agencies that specify that no annual budget funds will be ment generates trust. allocated until contracts have been evaluated and results monitored. 140 Ten Steps to a Results-Based Monitoring and Evaluation System In other cases, "If the agency contracts or provides grants to other organizations for services to customers, it can include outcome-based performance targets in the agreements and then compare outcomes against those targets" (Hatry 1999, p. 170). Rewards and penalties based on performance can also be delineated in such contracts. If there are no data on which to base decisions, those decisions can be arbitrary. At the same time, decisionmakers always have the dis- cretion to make their own decisions. However, better decisionmaking will result from taking the time to monitor, measure, and evaluate, and incorporate the findings into the decisionmaking process. An in- teresting corollary to this is that if one starts to ask for performance information, improved performance will result. Other uses of results findings include identifying best practices, supporting economies of scale, avoiding overlap and duplication, and coordinating similar programs across agencies (Wye 2002, p. 49). There are many examples of using findings. Boxes 9.2 and 9.3 il- lustrate some of the different uses of performance findings. Additional Benefits of Using Findings: Feedback, Knowledge, and Learning M&E systems provide important feedback about the progress, as well as the success or failure, of projects, programs, and policies throughout their respective cycles. These systems constitute a power- ful, continuous public management tool that decisionmakers can use to improve performance, and demonstrate accountability and trans- parency with respect to results. One way to consider M&E feedback within the development context is as follows: "Evaluation feedback has been broadly defined as a dynamic process which involves the presentation and dissemination of evaluation information in order to ensure its application into new or existing development activities . . . feedback, as distinct from dissemination of evaluation findings, is the process of ensuring that lessons learned are incorporated into new operations" (OECD 2001, p. 60). The use of M&E findings can promote knowledge and learning in governments and organizations. The new emphasis in the interna- tional aid community is more and more on local knowledge acquisi- tion, not knowledge transfer from donor to recipient. What exactly do we mean by "learning" in a results-based monitoring and evalua- tion context? "Learning has been described as a continuous dynamic process of investigation where the key elements are experience, Using the Findings 141 Box 9.2 Using Performance Data to Track and Reduce Crime in New York City Over the past decade, the New York City Police Department has used a special results-based M&E system to map the daily incidence of violent crime. "CompStat is a sophisticated performance measurement system that reorders an organization's day-to-day operations, as well as its overall orientation toward its core mission and goals. CompStat is based upon the compilation, distribution, and utilization of `real time' data in order to allow field managers to make better-informed and more effective deci- sions" (O'Connell 2001, p. 6). As former New York mayor Rudolph Giuliani noted, "We have 77 police precincts. Every single night they record all of the index crimes that have occurred in that precinct and a lot of other data. We record the number of civilian complaints. We record the number of arrests that are made for serious crimes and less serious crimes. It is all part of CompStat, a computer-driven program that helps ensure executive accountability. And the purpose of it is to see if crime is up or down, not just citywide, but neighborhood by neighborhood. And if crime is going up, it lets you do something about it now--not a year and a half from now when the FBI puts out crime statistics . . . Now we know about it today. And we can make strategic decisions accordingly" (O'Connell 2001, p. 9). As a result, during a five year period, "New York City experienced a precipitous drop in the burglary rate (53 percent), a 54 percent drop in reported robberies, and an incredible 67 percent drop in the murder rate . . . These extraordinary achievements were realized in large part due to the department's innovative model of police management, known as CompStat" (O'Connell 2001, p. 8). The overall result of using this real-time results-based system has been that "New York City now holds the undisputed title as the safest big city in the nation . . . " (NYC.gov 2003). Sources: O'Connell 2001, NYC.gov 2003. 142 Ten Steps to a Results-Based Monitoring and Evaluation System Box 9.3 U.S. Department of Labor--An Organization with a Mature, Functioning Results-Based M&E System The U.S. Department of Labor (DOL) is an example of an organization that has a mature, functioning results-based M&E system. Its efforts were jump-started by the U.S. Government Performance Results Act of 1993 (see box 10.2). The DOL established a mission, vision, strategic plan, and three main strategic goals: a prepared work- force; a secure workforce; and quality workplaces. Working from these three goals, the DOL then estab- lished three attendant outcomes for each of these larger strategic goals. Strategic Goal: l. A prepared workforce Outcomes: a. increase employment, earnings, and assistance b. increase the number of youth making a successful transition to work c. improve the effectiveness and information and analysis on the U.S. economy Strategic Goal: 2. A secure workforce Outcomes: a. increase compliance with worker protection laws b. protect worker benefits c. increase employment and earnings for retrained workers Strategic Goal: 3. Quality workplaces Outcomes: a. reduce workplace injuries, illnesses, and fatalities b. foster equal opportunity workplaces c. reduce exploitation of child labor, protect the basic rights of workers, and strengthen labor markets Annual budgets are assigned for each of these strategic goals, and are later measured against actual budgetary outlays. The DOL then holds biannual reviews on each of these goals and includes the following information: Results: The most recent results available for the performance outcome Indicator: The measures that will be used to assess progress toward performance goal accomplishment Data Source: The measurement systems that will be used to collect performance indicator data Baseline: The baseline year and baseline level against which progress will be evaluated Comment: Issues related to goal accomplishment, measurement systems, and strategies that provide a context or description of the performance goal or indicator. Source: U.S. Department of Labor 2002. Using the Findings 143 knowledge, access and relevance. It requires a culture of inquiry and "A monitoring and evalua- investigation, rather than one of response and reporting" (UNDP tion framework that gener- 2002, p. 77). ates knowledge, promotes Knowledge and knowledge management are additional key com- learning and guides action ponents of using performance findings. New knowledge can be gen- is, in its own right, an im- erated through the use of findings on a continuous basis. Knowledge portant means of capacity management means capturing findings, institutionalizing learning, development and sustain- and organizing the wealth of information produced continually by ability of national results." the M&E system. Results-based monitoring and evaluation systems and units have a (UNDP 2002, p. 76) special capacity to add to the learning and knowledge process. When used effectively, M&E systems can be an institutionalized form of learning and knowledge. "Learning must therefore be incorporated into the overall programming cycle through an effective feedback sys- tem. Information must be disseminated and available to potential users in order to become applied knowledge . . . Learning is also a key tool for management and, as such, the strategy for the applica- tion of evaluative knowledge is an important means of advancing to- ward outcomes . . . Outcomes present more variables around which learning can and must take place" (UNDP 2002, pp. 75­76). Institutionalizing learning is important in governments and organ- izations. Policy and program evaluation should play a systematic instead of an ad hoc role in the process of organizational learning. A political environment needs to be created that encourages continuous reporting, as well as the use of results. This implies that a certain level of institutionalization has to occur before findings can be used in the management of government institutions. Emphasizing organizational learning as a means of enhancing organizational performance is a fruitful and promising area of engagement with the public sector. Box 9.4 provides an example of how German aid agencies are moving increasingly in the direction of evaluation-based learning. Many governments and organizations may yet be resistant to learn- ing, internalizing, and sharing performance findings within and be- tween ministries, organizations, agencies, and departments. There are a number of organizational, behavioral, and political challenges to be recognized. In box 9.5 we look at some of the obstacles to learning. Good M&E systems can help to overcome these obstacles to learn- ing. By producing a continual flow of feedback and data, M&E sys- tems help decisionmakers manage more effectively. Organizational 144 Ten Steps to a Results-Based Monitoring and Evaluation System Box 9.4 Signs of Improving Conditions for Evaluation-Based Learning in German Aid Agencies · Germany's diversified development co-operation structure is now gradually moving towards greater concentration on particular issues, priority areas and countries. There is also a parallel trend towards greater decentralisation. · German official aid agencies see themselves more than ever as learning organisations, and are begin- ning to restructure their management systems accordingly. Evaluation systems are intended to play a key part in this, and are being given greater priority and greater institutional independence. · The quality of evaluation is improving. More sophisticated methods, more impact orientation and a greater number of broader-based evaluations (not confined to a single project) all offer the prospect that in future more of the knowledge will be generated that is needed for both quality improvement and con- ceptual advancement of development cooperation work, and for greater external accountability. · Aid agencies themselves believe it is important to increase the extent to which they systematize and institutionalize their feedback system for evaluation-based learning and accountability." · Aid agencies see a strong need to do more to promote the internalization of evaluation lessons, tak- ing a more systematic and innovative approach. Some are currently appraising the inclusion of this in an overall system of knowledge management. · . . . a substantial boost [has been given] to horizontal learning among German aid agencies in recent years. Source: OECD 2001, p. 19. cultures can be transformed through the use of M&E systems. There may be decreased pressures to spend as governments receive data that help them manage resource flows. M&E systems also provide built-in incentives to learn, pointing out directions, trends, successes, and problems. Tunnel vision can be overcome as data on results shed light on areas previously unknown or not fully understood. The loss of insti- tutional memory due to staff changes can also be minimized because M&E systems, when well maintained, produce a record of data over time. Finally, change can be managed more easily with continuous feedback. Obstacles can also be overcome by understanding how governments and organizations learn and by identifying and overcoming the im- pediments. There are ways to encourage greater use of performance findings through learning and knowledge building among govern- ments and organizations (box 9.6). Using the Findings 145 Box 9.5 Obstacles to Learning The OECD has identified several obstacles that can prevent learning: Organisational culture--some organisations have a culture where ac- countability tends to be associated with blame. This has the effect of dis- couraging openness and learning. In other [organizations], it is more ac- ceptable to own up to mistakes and see these as opportunities for learning, recognizing that there is often as much to learn from poorly per- forming projects as there is from success stories. Pressure to spend--learning takes time, and pressure to meet disburse- ment targets can lead to shortcuts being taken during project planning and approval stages, with lessons from previous experience being ignored or only selectively applied in the haste to get decisions through. Lack of incentives to learn--unless there is proper accountability . . . built into the project cycle there may be little incentive to learn. This is particularly the case when staff or consultants shift from task to task, and have generally moved on long before the consequences of failure to learn are felt. Tunnel vision--the tendency of some staff or operational units to get stuck in a rut, carrying on with what they know, even when the short- comings of the old familiar approaches are widely accepted. Loss of institutional memory--caused by frequent staff rotation or heavy reliance on short-term consultants, or by the weakening or dis- banding of specialist departments. Insecurity and the pace of change--if staff are insecure or unclear about what their objectives are, or if the departmental priorities are fre- quently shifting, this can have an adverse effect on learning. The unequal nature of the aid relationship--which tends to put donors in the driving seat, thereby inhibiting real partnerships and two- way knowledge sharing. Source: OECD 2001, pp. 20­21. 146 Ten Steps to a Results-Based Monitoring and Evaluation System Box 9.6 Incentives for Learning, Knowledge Building, and Greater Use of Performance Findings Governments and organizations can pro-actively encourage staff to learn, build knowledge, and use performance findings. Here are just a few examples: · Develop guidance materials on the use of outcome information. · Provide training in uses of outcome information for managers and other staff who can use outcome information. · Hold regular `How are we doing?' sessions with staff soon after each outcome report becomes available. · Identify and reward offices, grantees, and facilities with good outcomes. · Develop grant allocation guidelines that reward improved performance. · Use the outcome data to identify successful (`best') practices within the agency . . . · Use outcome data to identify common problems, and if possible, solutions. · Use outcome information to identify needs for training for staff or technical assistance . . . · Use outcome information to help prioritize use of resources. Source: Hatry, Morley, Rossman, and Wholey 2003, pp. 16­17. Strategies for Sharing Information "Plan for communication as part of your M&E system from the out- set" (IFAD 2002 pp. 6­7). A good communication strategy is essen- tial for disseminating information and sharing it with key stakehold- ers. Results-based information should be shared with all internal and external stakeholders and interested parties. "Active follow-up [em- phasis added] is necessary to implement recommendations . . . and to incorporate lessons learned in future decision-making processes . . . The more stakeholders are involved in planning the next steps, the more likely they are to follow through on implementing evaluation recommendations" (UNPF 2002). Information sharing strategies de- signed for and targeted to specific stakeholder groups can also be help- ful. In this context, it helps to "[t]ry to adapt existing reporting re- quirements and resources to new uses and formats" (Wye 2002, p. 55). Using results information can take passive and active forms (box 9.7). Understanding the target audience is key. Communication strategies need to be tailored to suit a particular target audience--parliament, ministers, the media, the private sector, NGOs and civil society or- ganizations, and the general public. "Disclosure of negative or con- troversial evaluation findings can obviously create difficulties for Using the Findings 147 Box 9.7 Active and Passive Approaches to Using Results Information It is imperative that results information be used. Simply providing information to potential users within the government--managers and oversight agencies--is not enough. Even improved transparency through publi- cation of performance information is not enough. Countries have used different approaches to providing such an imperative, and generally fall into either active or passive groupings. More active measures include formal reviews (regularly scheduled meetings at which performance is assessed), senior management attention (either as the chair of the formal review or di- rect engagement in monitoring and following up on performance exceptions), nonmonetary rewards (gener- ally public recognition with an award or honor). Many of these are blended for greater impact. Former U.S. Vice President Al Gore's "High-Impact Agency Initiative," and the U.K. Prime Minister's Office's six month performance reviews, are examples of active approaches. Passive approaches include performance contracts (formal agreements between managers and staff on targets, implying a formal review at the end of the contract period), peer pressure (a scorecard of per- formance for each unit, made widely available so the units can be easily compared), public embarrassment or approval, or monetary incentives (hope of monetary benefit if performance improves or targets are achieved, either on an individual basis by tying senior management pay or bonuses to organizational per- formance, or on an overall basis by tying organization-wide pay or bonuses to organizational performance, or by trying to link the organization's budget to its performance). These are "passive" insofar as they set up a structure, but do not ensure the performance measures are used to affect decisions. Source: Dorotinsky 2003a. agencies . . . But . . . the benefits of disclosure in the long run make it "Performance information worthwhile . . . Greater disclosure can also increase the pressure for can make a dramatic con- more systematic follow-up of recommendations, while motivating tribution to improving gov- those involved in evaluations to produce a better product, since they ernment performance if it is know their report will be made public, rather than being buried on a effectively communicated shelf somewhere" (OECD 2001, p. 26). to stakeholders, including Governments and organizations can use a wide array of strategies citizens." for sharing information with internal and external stakeholders. These strategies also involve a number of different media that can be (Wye 2002, p. 53) used to share the performance information. Empower the Media The media can be an important partner in disseminating the findings generated by results-based M&E systems. For example, the media 148 Ten Steps to a Results-Based Monitoring and Evaluation System often report on whether governments or organizations have actually delivered on promised projects, programs, policies, and services. The media have also been instrumental in exposing corruption and calling for good or better governance in many countries. Enact "Freedom of Information" Legislation Freedom of information is another powerful tool that can be used to share information with concerned stakeholders. For example, the government of Romania enacted freedom of information legislation recently with the stipulation that, except for information that could impair the country's ability to protect and defend itself, anyone who asks for information about how well the government is performing will receive it (World Bank 2001d). Institute E-Government E-government is increasingly being used as a tool by governments around the world, and has become a particular priority among OECD countries. E-government involves the use of information tech- nology to provide better accessibility, outreach, information, and services. It represents a new electronic environment in which stake- holders can interact directly with the government, obtain information from the government, and even transact business online. Developing countries are moving in this direction, too. The government of Jor- dan, for example, is beginning its e-government initiative with the in- troduction of electronic procurement and accounting. Put Information on Internal and External Internet Sites The use of internal (agency or government) and external Web sites that include published performance findings is yet another effective way of sharing information. Many agencies are also developing searchable databases for M&E findings. Publish Annual Budget Reports There is no more important way to communicate how taxpayer money is being spent than to publish the budget. Citizens will have the opportunity to "compare" the quality and level of services being provided by the government, and the priority of that service or pro- gram in the expenditure plan. Engage Civil Society and Citizen Groups Engaging civil society and citizens groups also involves the inclusion of " . . . accountability, advocacy and action-oriented audiences Using the Findings 149 and . . . agree[ment] on the information (content and form) they need" (IFAD 2002, p. 6-6). Strengthen Parliamentary Oversight Strengthening parliamentary oversight is another important way to share and disseminate information. Many parliaments have active budget or public accounts committees in lower or upper chambers. There are also other agencies that provide parliaments with over- sight, for example, the U.S. General Accounting Office (GAO), the audit and evaluation office of the Congress, or the National Audit Of- fice for the Parliament in the U.K. The GAO and similar government organizations and agencies also perform an investigative function for the parliaments they serve. Parliaments in various countries--both de- veloped and developing--are starting to ask for performance informa- tion as part of their oversight function (see box 9.8). They are looking to see that budgets are used effectively; thus, more governments are considering moving toward programmatic budgeting. Strengthen the Office of the Auditor General Many countries are also finding the Office of the Auditor General to be a key partner in determining whether governments are functioning effectively. Interestingly, as audit agencies demand more information about how well the public sector is performing and how projects, programs, and policies are actually being implemented, we are start- ing to see better implementation. In Canada, the Treasury Board produced a "Guide for the Devel- opment of Results-Based Management and Accountability Frame- Box 9.8 Canadian Government Performance Reports to Parliament "Each year since 1997 the government has tabled two sets of departmental reports in Parliament. In the spring, departments and agencies produce their Reports on Plans and Priorities for the coming fiscal year. In the fall they provide parliamentarians with their Departmental Performance reports indicating achievements attained over the previous fiscal year." An annual report, "Canada's Performance," is produced for the Parliament. It contains 19 societal indicators grouped into four main themes: economic opportunities and innovation in Canada; the health of Canadians; the Canadian environment; and the strength and safety of Canadian communities. Parliamentarians have emphasized the need for such indicators of results findings to be relevant, temporal, available, comparable, and understandable. Source: President of the Treasury Board of Canada 2002, pp. 2­3. 150 Ten Steps to a Results-Based Monitoring and Evaluation System works." It is " . . . intended to serve as a blueprint for managers to help them focus on measuring and reporting on outcomes through- out the life cycle of a policy, program or initiative" (Treasury Board Secretariat of Canada 2001, p. 1). Share and Compare Results Findings with Development Partners Sharing and comparing results findings with development partners is also beneficial on a number of levels. " . . . [L]earning from evaluative knowledge becomes wider than simply organizational learning and also encompasses development learning. It helps to test systematically the validity, relevance and progress of the development hypotheses" (UNDP 2002, p. 76). Since the introduction of National Poverty Re- duction Strategies and similar broadly based strategies and policies, the need for information sharing among development partners--es- pecially bilateral and multilateral aid agencies--has increased. "These and other joint initiatives are premised on the assumption that coordinated agency action will be more effective than individual efforts. Yet mechanisms for exchanging evaluation lessons between [aid] agencies are still weak, and practical hurdles continue to get in the way of more frequent joint evaluations--which, when they do occur, are generally seen as a very good way of sharing lessons and methodologies" (OECD 2001, p. 31). More could also be done with respect to sharing performance findings with donor recipient coun- tries. All key stakeholders--particularly recipient countries--need to be part of the M&E process from start to finish. There are many uses for performance findings. We looked at two successful examples involving crime information and a government organization with a mature, functioning M&E system. We also ex- amined the many benefits of using findings, including continuous feedback, and organizational and institutional learning and knowl- edge. We acknowledged and examined the obstacles and incentives-- many of them political--to using findings, and looked at some poten- tial strategies for sharing information among internal and external stakeholders. We turn now in the next chapter to the final step of our model on sustaining the results-based M&E system within your organization. Chapter 10 Step 10: Sustaining the M&E System within the Organization Figure 10.1 Selecting Key Planning for Conducting Indicators to Improvement -- a Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 6 7 8 9 10 10 Agreeing on Baseline Data Monitoring Reporting Sustaining Outcomes to on Indicators -- for Results Findings the M&E Monitor and Where Are We System Evaluate Today? within the Organization In the final step of our model, we turn to sustaining results-based M&E systems. An M&E system should be regarded as a long-term effort, as opposed to an episodic effort for a short period or for the duration of a specific project, program, or policy. Sustaining such systems within governments or organizations recognizes the long- term process involved in ensuring utility (for without utility, there is no logic for having such a system). Specifically, we will examine: (a) six critical components of sustaining results-based M&E systems; (b) the importance of incentives and disincentives in sustaining M&E systems; (c) possible hurdles in sustaining a results-based M&E sys- tem; (d) validating and evaluating M&E systems and information; and (e) M&E stimulating positive cultural change in governments and organizations. 151 152 Ten Steps to a Results-Based Monitoring and Evaluation System Six Critical Components of Sustaining Results-Based M&E Systems We will examine six critical components involved in building the sus- tainability of M&E systems. Each of these dimensions needs continu- ous attention and care. Demand If demand is episodic or haphazard, results-based M&E systems are not going to be used and sustained. Structured requirements for re- porting results, including legislation, regulations, and international development requirements (HIPC and EU accession, for example), can help lead to sustained, consistent demand for such systems. Gov- ernments, civil society, and donors are increasingly requiring the re- sults that M&E systems can best track, monitor, and measure. In many cases, demand can also be stimulated when the strategic Sustainability and use of goals of the government are translated into results-based M&E sys- M&E systems are interde- tems, such as through National Poverty Reduction Strategies and pendent. Systems that are other initiatives. These are not simply activity-driven initiatives; not used will not be sus- rather, they try to answer the "so what" question. What are the con- tainable. The issue of use sequences of policy and program efforts to reduce poverty and ad- has to be addressed first. It dress the most vulnerable groups? is the prerequisite to system sustainability. Clear Roles and Responsibilities Clear roles and responsibilities and formal organizational and politi- cal lines of authority must be established. The organization and people who will be in charge of collecting, analyzing, and reporting performance information must be clearly defined. Guidance is neces- sary. For example, a Ministry of Finance may be responsible for ad- ministering National Poverty Reduction Strategies or initiatives, and will need to issue directions to the sector or line ministries to collect and report on data relevant to tracking the various outcomes speci- fied in the strategy. Internal political coordination is key. A system should be built that links the central planning and finance ministries to the line and sector ministries. These bridges linking ministries are important, as is the need for horizontal communication to keep all concerned parties in- formed. If there are organizational problems, these should be dealt with sooner rather than later. It is also important to build a continuous system of data collection and analysis that goes beyond the national government to other lev- Sustaining the M&E System within the Organization 153 els of government. Data collection, analysis, and reporting should be aligned throughout the various levels of government. For example, in the health or education sectors, focusing at the local and regional lev- els will be important because some of the requirements to meet na- tional goals are going to take place there. Data analysis and reporting at these levels will then feed into the larger national data base in de- termining progress toward the desired outcomes. Finally, M&E systems should be built in such a way that there is a demand for results information at every level that data are collected and analyzed. There is no level of the system that is a mere "pass through" of information. Pass-through parts of the system create tremendous vulnerability, and can lead to breakdowns in M&E sys- tems. If people are not involved, if there is no ownership, then people in the "pass-through" levels will begin to lose interest and the result will be poor data collection and reporting. Trustworthy and Credible Information The M&E system must be able to produce results information that brings both good and bad news. Performance information should be transparent and made available to all key stakeholders. If debate of issues is not backed up by trustworthy and credible information, only personal opinions and presumptions are left. It should also be noted that the producers of results information need protection from political reprisals. If bad news brings career problems to the messengers, fear will permeate the system and the reliability of the information produced will be compromised. A quick way to un- dermine an M&E system is to punish those who deliver bad news. Information produced by the M&E system should be transparent and subject to independent verification. If data on government per- formance are held too close, or there are gatekeepers who prevent the release of such information, the system will again be faulty. As a fur- ther check on the system, it would be advisable to have a periodic in- dependent review by the national audit office, parliament, or a group of academics to ensure that the data being generated by the system are accurate and reliable, and to build confidence among managers who could use the data. Accountability No part of the government should be exempt from accountability to stakeholders. Civil society organizations and NGOs (such as Trans- parency International) can play a key role in encouraging trans- 154 Ten Steps to a Results-Based Monitoring and Evaluation System parency and accountability, and can even help with collecting data. For example, NGOs in Bangladesh help to collect local educational data because the capacity to collect and report on such data is very weak within the government. The media, private sector, and parlia- ment also have roles to ensure that the information produced is timely, accurate, available, and addresses government performance. It is also important not to reward failure. Accountability means that problems should be acknowledged and addressed. Capacity Sound technical skills in data collection and analysis are necessary for the system's sustainability. Managerial skills in strategic goal set- ting and organizational development are also needed. Data collection and retrieval systems must be up and running--and modernized. Governments will need to commit continuing financial resources to the upkeep and management of results-based M&E systems. Institu- tional experience and memory are also helpful in the long-term sus- tainability of these systems. Incentives Incentives need to be introduced to encourage use of performance in- formation. This means that success needs to be acknowledged and re- warded, problems need to be addressed, messengers must not be pun- ished, organizational learning is valued, and budget savings are shared. Corrupt or ineffective systems cannot be counted on to pro- duce quality information and analysis. Examples of the ways that governments have sought to incorpo- rate and sustain results-based environments are provided in boxes 10.1 and 10.2, the U.K. Citizen's Charters and the U.S. Government Performance Results Act (GPRA), respectively, incorporate the criti- cal sustainability components. The Citizen's Charter is relevant to the sustainability of M&E in that it establishes an ongoing government­ citizen contract outlining responsibilities and performance expecta- tions. The U.S. GPRA also legally institutionalizes M&E within gov- ernment agencies, making such systems sustainable in the longer term. Developing countries are also working toward creation of evalua- tion capacity, institutionalization of evaluation, and use of results findings within government--in short, sustainable M&E systems. Table 10.1 provides a comparative illustration of such efforts in Colombia, China, and Indonesia. Sustaining the M&E System within the Organization 155 Box 10.1 Citizen's Charters in the United Kingdom In the U.K., there are contracts signed between the government and citizen groups, called "Citizen's Char- ters," that specify that the government will be accountable to the public for a certain level of performance. The Citizen's Charter, launched in 1991, aims to improve public services and make the sevices more re- sponsive to users. There are now 40 main charters that cover key public services and set out the standards of service people can expect to receive. There are also over 10,000 local charters covering local service providers, such as schools, police forces, and fire services. In addition, Charter Quality Networks were relaunched as a network of managers from public services to exchange ideas on the charter program, customer service, and quality issues and to share best practice. There are now 22 Quality Networks around the U.K., involving over 1,500 people. As part of the Better Government initiative, the Charter Unit has set up a People's Panel of around 5,000 people across the U.K. The panel is being used to consult members of the public on their attitudes toward public services and generate ideas about how services can be improved. In addition to the Cabinet Office, other departments, agencies, and public bodies use the panel for research and consultation. Source: U.K. Cabinet Office. The Importance of Incentives and Disincentives in Sustaining M&E Systems Sustaining M&E systems also involves using appropriate incentives to keep managers and stakeholders on track and motivated. "Putting in place incentives for M&E means offering stimuli that encourage . . . M&E officers and primary stakeholders to perceive the usefulness of M&E, not as a bureaucratic task, but as an opportunity to discuss problems openly, reflect critically and criticize constructively in order to learn what changes are needed to enhance impact" (IFAD 2002, Section 7 p. 4). There are a variety of organizational, financial, resource, politi- cal, technical assistance, and training incentives that can be used to sus- tain M&E systems. Likewise, managers need to remove disincentives to sustaining M&E systems. Boxes 10.3 and 10.4 contain checklists of the kinds of incentives and disincentives that should be considered. Possible Problems in Sustaining Results-Based M&E Systems There are a number of hurdles that may arise in sustaining M&E sys- tems. Hatry (1999) brings to light a number of likely problems in im- plementing and sustaining M&E systems, as follow: 156 Ten Steps to a Results-Based Monitoring and Evaluation System Box 10.2 U.S. Government Performance and Results Act of 1993 Performance measurement in the U.S. began first with local governments in the 1970s, spread to state gov- ernments, and eventually to the federal level with the enactment of the Government Performance and Re- sults Act (GPRA) in 1993. The U.S. federal government adopted a performance measurement system later than other levels of American government, and actually later than some foreign governments. "The purposes of the [U.S. Government Performance and Results] Act are to: (1) improve the confidence of the American people in the capability of the Federal Government, by systematically holding Federal agencies accountable for achieving program results; (2) initiate program performance reform with a series of pilot projects in setting program goals, measuring program performance against those goals, and report- ing publicly on their progress; (3) improve Federal program effectiveness and public accountability by pro- moting a new focus on results, service quality, and customer satisfaction; (4) help Federal managers im- prove service delivery, by requiring that they plan for meeting program objectives and by providing them with information about program results and service quality; (5) improve congressional decision-making by providing more objective information on achieving statutory objectives, and on the relative effectiveness and efficacy of Federal programs and spending; and (6) improve internal management of the Federal Gov- ernment" (U.S. Office of Management and Budget 1993). A recent survey of 16 programs across 12 U.S. government agencies found that "[m]any federal programs have already made use of regularly collected outcome data to help them improve their programs . . . Federal managers have used outcome data in a variety of ways, [including] to trigger corrective action; identify and encourage `best practices'; motivate [and recognize staff]; and plan and budget . . . " At the same time, the survey found some continuing obstacles--indeed obstacles that can affect any organization--to the use of outcome data: (a) lack of authority or interest to make changes; (b) limited understanding of use of out- come data; (c) outcome data problems (such as old data, nondisaggregated data, lack of specificity, need for intermediate data, and so forth); and (d) fear of "rocking the boat" (Hatry, Morley, Rossman, and Wholey 2003, pp. 11­13). Most recently, GPRA has been extended to the integration of the performance and budget areas. Efforts are also being made across the government to group GPRA strategic and annual planning and reporting more closely. "Overall GPRA is just `good business.' Its requirements have provided government Departments with tools for very basic ways of conducting business in sensible ways: set performance goals and measure both long and short-term outcomes. Any organization seeking to provide improved quality of life, greater quan- tity of services, and enhanced overall quality of customer services must have a vision and a mission, set goals and objectives, and must measure results" (ChannahSorah 2003, pp. 5­6.) Sustaining the M&E System within the Organization 157 Table 10.1 Evaluation Capacity Development and Institutionalization--Key Issues Addressed in Colombia, China, and Indonesia Issue Colombia China Indonesia Anchoring the Constitution mandates State Council draft Responsibility rests with evaluation regime the Executive to take resolution calls on the the Executive through the lead. Central Executive Agencies a Ministerial decree. to take lead. Positioning the Centralized in the Decentralized in key Centralized in the evaluation function National Planning central agencies. National Development Departmentt (NPD). Key Planning Agency line agencies provide inputs. (BAPPENAS). Line agencies provide inputs. Evaluation coverage Public policy and major Public sector projects. Development policies, public sector programs. plans, programs, and projects. Linking evaluation NPD plays a key role in No formal links have BAPPENAS to link with other public policy and strategy been established. State evaluation to the annual sector functions formulation and budget Planning Commission budget allocation process. allocation and monitoring. involved in public resources allocation and monitoring. Using evaluation in Monitoring and Monitoring and Monitoring and evaluation decisionmaking evaluation information evaluation to inform information to flow through to flow to line agency central agency through line agency heads and the NPD. management. management to BAPPENAS. Professionalizing the Evaluation is a trans- Evaluation is seen Evaluation is not seen as evaluation function discipline cutting across primarily as applied a separate profession, specific professional skills. socioeconomic analysis. but a complementary discipline. Resources for Evaluation to be main- Evaluation mainstreamed Evaluation mainstreamed evaluation streamed in agencies' in central agencies' in agencies' budgets. budgets. budgets. Note: BAPPENAS = Badan Perencanaan Pembangunan Nasional; NPD = National Planning Department. Source: Guerrero 1999, p. 180. 158 Ten Steps to a Results-Based Monitoring and Evaluation System Box 10.3 Checklist for Staff Incentives That Encourage Learning-Oriented, Participatory M&E Are the following incentives in place? · Clarity of M&E responsibility · Financial and other physical rewards: appropriate salaries and other rewards · Activity support: support, such as financial and other resources, for carrying out project, program, or policy activities · Personnel and partner strategy: hiring staff who have an open attitude to learning, and signing on partners who are willing to try out more participatory forms of M&E · Project, program, or policy culture: compliments and encouragement for those who ask questions and innovate, giving relatively high status to M&E among staff · Performance appraisal processes: equal focus on staff capacity to learn and innovate, rather than just if they have reached their quantitative targets · Showing the use of M&E data: making the data explicit and interesting by displaying them · Feedback: telling data collectors, information providers, and others involved in the process how their data were used (analyzed), and what it contributed to the project. Source: IFAD 2002. Box 10.4 Checklist for Staff Disincentives That Hinder Learning-Oriented, Participatory M&E Have the following disincentives been removed from project, program, or policy? · Using the M&E unit as the place to park demoted or unqualified staff · Not making clear how data will be or were used · Chastising those who innovate within their project boundaries or those who make mistakes · Focusing performance appraisals only on activities undertaken (outputs) · Frequent rotation of staff to different posts · Staff feeling isolated or helpless in terms of their contribution being recognized toward achieving the project goal (the "line of sight" issue) · Unconstructive attitudes toward what constitutes participation or toward the primary stakeholder groups. Source: IFAD 2002. Sustaining the M&E System within the Organization 159 · Personnel training needs · Overall system cost and feasibility · Changes in legislative and agency priorities · Maintaining indicator stability over time · Documentation of the outcome measurement process (who will do what) · Fear and resistance from program managers · Participation by other levels of government and the private sector · Aggregation of outcomes across projects, programs, or sites · Community-wide versus program-specific outcomes · Legislative support · Politics. Some of the most critical issues in implementing and sustaining M&E systems are the challenges in the human resource area. These challenges are perhaps not so different from all public sector human resource matters, but there are unique dimensions that have to be ad- dressed. First, there are issues in recruiting and holding talented staff who can build and manage a new information system. Can they be found and, if so, can they be hired? Second is the issue of what staff will risk venturing into a new government initiative--or stated differ- ently, what is the caliber of those who leave their present positions for positions in a new M&E unit? Third is the matter of whether the first cohort of those hired are change agents. Building an M&E sys- tem is a politically charged change process. Do those being hired un- derstand this and are they ready to manage a change process? Fourth, can continuous training be provided for all personnel at all levels? New methodologies, technologies, and procedures are in- evitable and need to be shared with staff. Can that training be pro- vided? Furthermore, given staff turnover, how soon and how ade- quately can new staff be trained to quickly increase their productivity and contributions to the unit? The M&E system will have to respond and adapt to changes in legislative and organizational priorities. In spite of these larger politi- cal and environmental changes, maintaining indicator stability over time is important. One wants to be able to compare similar issues and trends over a given period of time. 160 Ten Steps to a Results-Based Monitoring and Evaluation System Validating and Evaluating M&E Systems and Information Continued upgrading and improvement is important in sustaining re- sults-based M&E systems. M&E systems themselves should be evalu- ated periodically, using internal or external evaluators. "Evaluators can assist in validating performance data and improving performance measurement systems. Evaluations of performance measurement sys- tems should focus both on the technical quality of the measurement system and on the extent to which performance information is used in managing to achieve performance goals and in providing account- ability to key stakeholders and the public" (Wholey 2001, p. 345). Evaluators can also verify and confirm the results of M&E systems. M&E: Stimulating Positive Cultural Change in Governments and Organizations M&E systems are essentially political challenges, and to a lesser ex- tent, technical ones. Creating, implementing, and sustaining results- based M&E systems can help to bring about major cultural changes in the way governments and organizations operate. M&E systems can bring about positive cultural changes that lead to improved per- formance, enhanced accountability and transparency, and learning and knowledge (see box 10.5). Good results-based M&E systems must be used to be sustainable. Six components are necessary in sustaining these systems: demand, incentives, clear roles and responsibilities, trustworthy and credible information, accountability, and capacity. Sustainable M&E systems do exist in many OECD countries, and some developing countries are on their way toward building and sustaining such systems as well. Above all, results-based M&E systems are powerful public management tools that facilitate positive cultural and political changes in governments and organizations to demonstrate results, ac- countability, and transparency. They also facilitate knowledge and learning. And, they are doable! Last Reminders · The demand for capacity building never ends. The only way an organization can coast is downhill. · Keep champions on your side and help them. · Establish the understanding with the Ministry of Finance and the parliament that an M&E system needs sustained resources. Sustaining the M&E System within the Organization 161 Box 10.5 An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity A recent study examined the development of an evaluation culture in five different U.S. government agen- cies: the Administration for Children and Families, the Coast Guard, the Department of Housing and Urban Development, the National Highway Traffic Safety Administration, and the National Science Foun- dation. The five agencies used various strategies to develop and improve evaluation. Agency evaluation cul- ture, an institutional commitment to learning from evaluation, was developed to support policy debates and demands for accountability. The study found that key elements of evaluation capacity were an evaluation culture, data quality, ana- lytic experience, and collaborative partnerships. Agencies demonstrated an evaluation culture through regu- larly evaluating how well programs were working. Managers valued and used this information to test out new initiatives or assess progress toward agency goals. Agencies emphasized access to data that were credi- ble, reliable, and consistent across jurisdictions to ensure that evaluation findings were trustworthy. Agencies also needed access to analytic experience and research expertise. Finally, agencies formed collaborations with program partners and others to leverage resources and expertise to obtain performance information. Source: U.S. GAO 2003. · Look for every opportunity to link results information to budget and resource allocation decisions. · Begin with pilot efforts to demonstrate effective results-based monitoring. Begin with an enclave strategy (that is, islands of in- novation) as opposed to a whole-of-government approach. · Monitor both implementation progress and results achievements. · Complement performance monitoring with evaluations to ensure better understanding of public sector results. Chapter 11 Making Results-Based M&E Work for You and Your Organization Why Results-Based M&E? Results-based M&E has become a global phenomenon as national and international stakeholders in the development process have sought increased accountability, transparency, and results from gov- ernments and organizations. Multilateral development institutions, donor governments, parliaments, the private sector, NGOs, citizens' groups, and civil society are all voicing their interest in and concern for tangible results. Political and financial support for governments and their programs are becoming increasingly linked with a govern- ment's ability to implement good policies, demonstrate effectiveness in the use of resources, and deliver real results. The MDGs, the HIPC initiative, IDA funding, WTO membership, and EU accession are examples of just some of the international ini- tiatives and forces for change in the direction of results-based M&E. Internally, governments are facing the challenges of deregulation, commercialization, and privatization, as well as fluctuating budgets and resources. 162 Making Results-Based M&E Work for You and Your Organization 163 For these reasons, governments and organizations are turning to results-based M&E in the hope that this public management tool can help them devise appropriate policies, manage financial and other re- sources, and fulfill their mandates and promises to internal and ex- ternal stakeholders. Results-based M&E moves beyond the traditional input­output- focused M&E, and, when used effectively, helps policymakers and decisionmakers focus on and analyze outcomes and impacts. After all, inputs and outputs tell little about the effectiveness of a given policy, program, or project. While traditional M&E remains an im- portant part of the chain of results-based M&E, it is the outcomes and impacts that are of most interest and import to governments and their stakeholders. Building and sustaining results-based M&E systems is admittedly not an easy task. It requires continuous commitment, champions, time, effort, and resources. There may be many organizational and technical challenges to overcome in building these systems. Political challenges are usually the most difficult. And it may take several at- tempts before the system can be tailored to suit a given governmental or organizational policy, program, or project. But it is doable. And it is certainly worthwhile in light of the increasingly common demands for and conditions attached to demonstrating good performance. Good M&E systems also build knowledge capital by enabling gov- ernments and organizations to develop a knowledge base of the types of policies, programs, and projects that are successful--and more generally, what works, what does not, and why. Results-based M&E systems also help promote greater transparency and accountability, and may have beneficial spill-over effects in other parts of a govern- ment or organization. In short, there is tremendous power in measur- ing performance. Many of the OECD countries have had 20 or more years of expe- rience in M&E, and are at varying stages of progress with regard to results-based M&E systems. The OECD countries--like their devel- oping country counterparts--created evaluation cultures and M&E systems in response to varying degrees of internal and external pres- sures. Furthermore, developed countries have chosen a variety of starting points for implementing results-based M&E systems, includ- ing whole-of-government, enclave, and mixed approaches. Recent OECD survey results found that most OECD member countries now include performance information in their budgets. 164 Ten Steps to a Results-Based Monitoring and Evaluation System With respect to results considerations, about half of the countries have taken into account the distinction between outputs and outcomes. Much remains to be done though, such as linking performance tar- gets to expenditures, and using performance information to deter- mine budgetary allocations. Thus, in many OECD countries, results- based M&E is still a work in progress. The lessons learned from the OECD countries' experiences are useful and applicable to developing countries as they now face the challenges of creating their own M&E systems and cultures. OECD countries with democratic political systems, strong empirical tradi- tions, civil servants trained in the social sciences, and high levels of expenditure on education, health, and social welfare have been among the most successful in adopting results-based M&E systems. In fact, building such systems is first and foremost a political activity with technical dimensions rather than vice versa. The OECD experi- ence demonstrates that creating results-based M&E systems requires continuous effort to achieve comprehensive coverage across govern- mental management and budgetary systems. Developing countries face a variety of unique challenges as they try to answer the "so what" question: What are the results and im- pacts of government actions and interventions? These countries may encounter such obstacles as lack of demand for and ownership of M&E systems, weak institutional capacity, lack of bureaucratic co- operation and coordination, lack of highly placed champions, weak or nonexistent legal and regulatory frameworks, a traditional M&E culture, lack of workforce capacity, political and administrative cul- tures not conducive to M&E implementation, and so forth. Despite these obstacles, many developing countries have made impressive progress in developing results-based M&E systems. The challenges are difficult, but good government is essential for achieving eco- nomic, social, and human development. Developing countries deserve good government no less than others. Finally, given the increasing number of internal and external part- nerships that are being formed to accomplish development goals, a new need has emerged for M&E systems that encompasses these broader partnership efforts. International coordination of results is the next stage in the evolutionary process of extending results-based M&E. Making Results-Based M&E Work for You and Your Organization 165 How to Create Results-Based M&E Systems The ten-step model presented here can help governments and organiza- tions create, develop, and sustain results-based M&E systems. This model may be used for policies, programs, and projects. Though visually it appears as a linear process, in reality it is not. One will inevitably move back and forth along the steps, or work on several steps simultaneously. The model has some unique features, including Step 1, conducting a readiness assessment. This assessment--often missed or omitted-- is a diagnostic tool that determines whether governments are actually ready and able to move forward in building, using, and sustaining M&E systems. The three main parts of the readiness assessment in- clude an examination of incentives or demands for designing and building a results-based M&E system, roles and responsibilities and existing structures for assessing performance of the government, and capacity building requirements. More specifically, the readiness as- sessment looks at eight key areas, including the following: what or who is encouraging the need for M&E systems; motivations of cham- pions; ownership and beneficiaries of systems; how the system will support better resource allocation and achievement of goals; dealing with negative or detrimental information generated by M&E; exist- ing capacity to support M&E systems; and links between the M&E system and project, program, sector, and national goals. A variety of lessons learned have already been generated by readi- ness assessments conducted in developing countries. For example, Bangladesh had few of the necessary requirements to begin building M&E systems. Assessments in Egypt and Romania, however, yielded vital information about likely entry points for beginning work on M&E. Highly placed political champions and strong, sustained polit- ical leadership were found to be key ingredients in the M&E mix. Other findings are that ministries may be at different stages in the ability to conduct M&E. It may be possible to move forward with M&E by working with pockets of innovation within government. Communication and coordination within and between government agencies and departments and among donors are also important. De- veloping countries may currently lack the institutional, human, and technical capacity to design, implement, and use results-based M&E systems; however, this is not an insurmountable obstacle. Training 166 Ten Steps to a Results-Based Monitoring and Evaluation System and technical assistance can be provided to remedy these difficulties. But no amount of training and technical assistance can substitute for indigenous political will. Often the political challenges are more diffi- cult to overcome than the technical ones. Choosing outcomes to monitor and evaluate is the second step. All governments must set goals, regardless of whether they have the capacity to conduct M&E. Outcomes will show which road to take. Building the M&E system is essentially a deductive process in which inputs, activities, outputs, and outcomes are all derived from the set- ting of longer term strategic goals. Likewise, setting outcomes is the first building block for developing a performance framework. Indica- tors, baselines, and targets will all flow from the outcomes. Building M&E systems is a participatory political process, and key internal and external stakeholders should be consulted during the various steps of the model--including the readiness assessment, the setting of outcomes, establishment of indicators, and so on. Critical stakeholders and their main concerns will need to be identified. Ex- isting problems need to be reformulated into a set of positive out- comes. Outcome statements need disaggregation, and each statement should contain only one goal. (This becomes important when devel- oping indicators and targets). Agreeing on strategic priorities and outcomes will then help drive resource allocation. Key performance indicators (Step 3) can only be set after agreeing upon and setting common goals. As with the case of outcomes, the interests of multiple stakeholders should be taken into account when selecting indicators. Indicators are the quantitative or qualitative variables that provide a simple and reliable means to measure achievement of goals. As stressed throughout the model, indicators should be developed for all levels of the results-based M&E system, meaning that indicators will be needed to monitor progress with re- spect to inputs, activities, outputs, outcomes, and impacts continu- ally. Progress needs to be monitored at all levels of the system to provide feedback on areas of success, as well as areas where improve- ments may be needed. Good performance indicators should be clear, relevant, economic, adequate, and monitorable ("CREAM"). Every indicator also needs it own separate M&E system, so caution should be exercised in set- ting too many indicators. Proxy and predesigned indicators may be adopted with full recognition of the pros and cons of using them. Constructing good indicators often takes more than one try; arriv- Making Results-Based M&E Work for You and Your Organization 167 ing at the final set of indicators will take time. Piloting of indicators is essential. Indicators should be well thought through. And they should not be changed very often--this can lead to chaos in the overall data collection system. It should also be remembered that performance indicators can be used to provide continuous feedback, and can pro- vide a wealth of performance information. Many developing countries are making progress in the performance indicator selection process. Baselines, Step 4, are derived from outcomes and indicators. A performance baseline is basically information--qualitative or quanti- tative--that provides data at the beginning of, or just prior to, the monitoring period. It is used as a starting point from which to moni- tor future performance. Or, stated somewhat differently, baselines are the first measurements of the indicators. The challenge is to obtain adequate baseline information on each of the performance indicators for each outcome. Eight key questions were outlined with respect to building baseline information: sources of data, data collection methods, who collects data, how often data are collected, cost and difficulty to collect data, who analyzes data, who reports data, and who uses data. Sources are who or what provide data--not the method of collecting data. Data sources may be primary or secondary. There are a variety of data collection methods along the contin- uum from informal and less structured to more structured and for- mal methods. Data collection methods include conversation with concerned individuals, community interviews, reviews of official records, key informant interviews and participant observation, focus group interviews, direct observations, questionnaires, one time sur- veys, panel surveys, census, and field experiments. Data collection strategies necessarily involve some tradeoffs with respect to cost, pre- cision, credibility, and timeliness. Establishing baseline data on indicators is crucial in determining current conditions and in measuring future performance. Subsequent measurements from the baseline will provide important directional or trend data, and can help decisionmakers determine whether they are on track with respect to their goals. Selecting results targets is Step 5. Targets are the interim steps on the way to a longer-term outcome. Again, a deductive reasoning process is involved, in which targets are based on outcomes, indica- tors, and baselines. Selecting targets should also entail a consultative, political, participatory process with key stakeholders. Targets can be 168 Ten Steps to a Results-Based Monitoring and Evaluation System determined by adding desired levels of improvement to baseline indi- cator levels (assuming a finite and expected level of inputs and activi- ties). Targets should be feasible given all of the resource (input) con- siderations. Each indicator is expected to have only one target over a specified time frame. Target setting is the final step in building the performance frame- work. The performance framework in turn becomes the basis for planning--with attendant implications for budgeting, resource allo- cation, staffing, and so forth. Performance frameworks have broad applicability and can be usefully employed as a format for National Poverty Reduction Strategies, project plans, programs, and policies. Monitoring for results, Step 6, entails both implementation moni- toring (means and strategies) and results monitoring. The key prin- ciples of building a monitoring system include recognizing the per- formance information needs at the policy, program, and project levels; the need for performance information to move both horizon- tally and vertically in the organization; identifying the demand for performance information at each level; and identifying the responsi- bilities at each level. The major criteria for collecting quality performance data are the reliability, validity, and timeliness of the data. Every monitoring sys- tem needs ownership, management, maintenance, and credibility. Monitoring for results also calls for data collection and analysis of performance data. There will be quality assurance challenges in building monitoring systems. These are to be expected, so it is impor- tant to pretest data collection instruments and procedures. Building the monitoring system framework means that each out- come will require an indicator, baseline, target, data collection strategy, data analysis, reporting plan, and identified users. Achieving results through partnership is essential. Means and strategies will need to be set by multiple partners. One must look be- yond one's own organizational unit when considering available in- puts. Partnerships may be created elsewhere in one's own organization, or even with other organizations inside or outside of government. Step 7 involves using evaluation information to support a results- based M&E system. Monitoring and evaluation are complementary, and both are needed in these systems. Evaluation information can be used for a variety of purposes: making resource allocation decisions; rethinking causality of problems; identifying emerging problems; sup- porting decisionmaking in selecting among competing alternatives; Making Results-Based M&E Work for You and Your Organization 169 supporting public sector reform; and so on. Evaluation information can also be relevant at all phases of a given policy, program, or project cycle. The timing of evaluations is another consideration. Evaluative infor- mation is essential when: (a) regular measurements of key indicators suggest a sharp divergence between planned and actual performance; (b) performance indicators consistently suggest weak or no results from an initiative; (c) resource allocations are being made across policies, programs, or projects; and (d) similar projects, programs, or policies are reporting divergent evidence of outcomes. There are seven different types of evaluation: performance logic chain, pre-implementation assessment, rapid appraisal, case study, meta-evaluation, impact evaluation, and process implementation. Each is appropriate to specific kinds of evaluation questions. Quality evaluations can be characterized by impartiality, usefulness, stake- holder involvement, value for money, feedback and dissemination, and technical adequacy. Reporting findings, Step 8, is a critical step in the process. Contin- uous performance data and findings should be used to help improve policies, programs, and projects. In analyzing and reporting data, the more data measurements there are, the more certain one can be of trends, directions, and results. There is an implicit tradeoff between measurement frequency and measurement precision. Cost and capac- ity also come into play. Performance data should be reported in comparison to earlier data and to the baseline. Also, to measure and compare against expected results, one must be able to compare present and past circumstances. Monitoring data are not causality data. They do not tell why an event occurred. It is also important to take into account the target audience when reporting findings. Using findings, Step 9, will better inform the decisionmaking process. There are a wide range of uses of performance findings. For example, performance-based budgets budget to outputs, but also help decisionmakers manage to outcomes. Another noteworthy phe- nomenon is that if performance information is asked for, improved performance will occur. Using continuous findings can also help to generate knowledge and learning within governments and organiza- tions. Building a credible knowledge management system is another key component of using findings. There are a variety of strategies that can be used to share informa- 170 Ten Steps to a Results-Based Monitoring and Evaluation System tion. A good communication strategy is essential for disseminating and sharing information with key stakeholders. Sharing information with stakeholders helps to bring them into the business of govern- ment and can help to generate trust. This is, after all, one of the pur- poses of building a results-based M&E system. Finally, Step 10 deals with sustaining the M&E system. We sug- gested there are six critical components to doing so: demand, clear roles and responsibilities, incentives, trustworthy and credible infor- mation, accountability, and capacity. We also examined the incen- tives and disincentives that may come into play in sustaining M&E systems. And we also know that problems will occur in implementing and sustaining the systems. Summing Up Results-based M&E systems are a powerful public management tool that can be used by governments and organizations to demonstrate accountability, transparency, and results. They can help to build and foster political and financial support and harmony for common poli- cies, programs, and projects. And they can help the government build a solid knowledge base. Importantly, results-based M&E systems can also bring about major political and cultural changes in the way governments and or- ganizations operate--leading to improved performance, enhanced ac- countability and transparency, learning, and knowledge. Results-based M&E systems should be considered a work in progress. Continuous attention, resources, and political commitment are needed to ensure the viability and sustainability of these systems. Building the cultural shift necessary to move an organization toward a results orientation takes time, commitment, and political will. In the absence of the efforts to undertake this transformation, the only way an organization can coast is downhill! Building and sustaining a results-based M&E system takes time and effort. No system is perfect, and there are many different ap- proaches, but the journey is worth the effort, and the rewards can be many. Annexes I­VI Annex I Assessing Performance-Based Monitoring and Evaluation Capacity: An Assessment Survey for Countries, Development Institutions, and Their Partners Introduction such an M&E system. With such information, the government, the donors, and partners can Countries across the globe are facing pressures then address the challenges of what training, to reform the policies and practices of their what organizational capacity building, and what public sectors. It is vital that an effective and sequencing of efforts will be needed to design efficient public sector contribute to sustainable and construct the necessary infrastructure to development, economic growth, and the well- produce, collect, analyze, and report relevant being of its citizens. Focusing on the perform- performance information. In short, it provides ance of the government thus becomes an impor- the basis for an action plan to move forward tant factor in being able to achieve the desired within the country. Furthermore, this survey can goals of growth and economic and social devel- help ensure that strategic goals are clearly opment. framed, that targets and baseline data are under- As governments begin to address these chal- stood as critical, and that the construction of lenges, they will want to document their results relevant indicators needs to be identified in the so as to provide credible and trustworthy infor- context of building the M&E system. mation both to their citizens and for their own The information is to be gathered from key management use. A results-based monitoring informants (government officials, members of and evaluation (M&E) system is an important civil society, NGOs, and so forth) in the country. tool that will allow governments to acquire this It is advised that the survey be administered in evidence. person by someone familiar with M&E capacity building as there are a number of open-ended The Survey questions where follow up and clarification This assessment survey is a diagnostic tool that questions will be useful. The survey consists of focuses on the current capacity of a government 40 questions and it is estimated (from the pilot) to design and build a results-based M&E sys- that it will take about 65 minutes to complete. tem. The intent is to learn what capacity and in- frastructure now exist and what new capacity Background Information: and infrastructure have to be built. The survey Name of Respondent:_______________________ is divided into three sections: Incentives; Roles Position:___________________________________ and Responsibilities; and Capacity Building. Organization:______________________________ The survey has been created as a tool to assist Years in Current Position: ___________________ individual governments, the donor community, Years in Current Organization:_______________ and their multiple development partners also in- Date of Interview: __________________________ volved in public sector reform to systematically Interview Conducted address the prerequisites (present or not) for By:_______________________________________ 174 Annex I 175 Part I: The Incentives For Designing and Build- lecting and using information on government ing a Performance-Based M&E System performance, for example, the Minister of Fi- 1. How would you describe the process of set- nance or Minister of Planning, the Minister ting priority goals and objectives in the cen- of Health, or Advisors to the President? tral ministries? In the sector or line ministries? 6. Are there senior officials that would resist 2. Can you identify any organizations that reg- requests for producing this kind of perform- ularly ask for information on how well the ance-based information? Reasons for the re- government is performing? sistance? - Ministry of Finance 7. Do any sector or line ministries undertake - Ministry of Planning or commission evaluations or formal re- - Prime Minister's Office views of the performance of projects, pro- - President's Office grams, or policies in their ministry? If so, - Individual Sector or Line Ministries which ones and what types of reviews? - Parliament -formal evaluations - Supreme (National) Audit Organization -client satisfaction surveys - Donors -performance audits - Private Sector -performance-based budget reviews - Media -other - NGOs 8. Are there formal requests from the parlia- - Citizens ment for information on the performance of 3. Does the Ministry of Finance or Ministry of the government to be supplied by the Min- Planning require any type of performance- istry of Finance or Ministry of Planning or based information on government projects, any of the sector ministries? programs, and policies be provided by the -for budget hearings sector ministries and other agencies in sub- -for parliament deliberations on the per- mitting their annual budget proposals? formance of government programs - Information on activities or outputs (ex- -for crafting of legislation pected from projects and programs) 9. Can you cite evidence of use by the parlia- - Information on outcomes or results ment of the performance information from (longer-term goals) the government: - Information from evaluations or other -for hearings formal reviews -for oversight of government performance - Expenditure data on priority goals for the -for the drafting of legislation government 10. Does the parliament have a "Public Accounts 4. Do any sector ministries or other agencies Committee" or a "Public Expenditure/Budget have requirements for reporting how well Committee?" If so, what are the functions of projects and programs are performing these committees? Do they use performance within their own organization? If so, which information as part of their activities? ones and what are the requirements? 11. Has civil society (media, NGOs, private sec- 5. Are there senior officials who advocate col- tor, and so forth) requested information on 176 Annex I government performance from the govern- sion of a system to track and report on the ment? If so, please describe. PRSP goals? 12. Has civil society published or broadcasted 22. Can you describe the status of the govern- any information on government perform- ment's efforts to implement an M&E system ance? If so, please describe. within their PRSP (and CDF, if relevant) 13. How easy (or not) has it been for members initiatives? of civil society to obtain information related to the performance of the government? Part II: Roles and Responsibilities for Assessing 14. Are NGOs or others in civil society collect- Performance of the Government ing data for their own use or as external 23. Are the regional and local levels of govern- monitors on how well the government is ment collecting information on their perform- performing? If so, please describe. ance to support budget expenditure decisions 15. Does any "freedom of information" legisla- or to enhance their program management? tion now exist? If not, is any such legisla- 24. Are there any evident links between the tion planned? Ministry of Finance and Ministry of Plan- 16. What information do the donors request of ning fiscal year budget allocations and the government on how well their individu- sector or line ministry performance? ally sponsored projects and programs are 25. Are there any formal roles or responsibili- performing? ties for civil society in the national govern- 17. Do the donors also ask for any other per- ment's planning processes? formance-based information from the gov- 26. Are there any formal roles or responsibili- ernment? If so, please describe. ties for civil society in the government's 18. How would you describe the audit function procedures for fiscal year budget allocation for the national government? Is there an in- decisions? dependent audit organization in the govern- 27. Is there any evident role for development ment and what is its function? Do indi- assistance or donor agencies in the national vidual ministries each have an internal audit planning process and setting of strategic function and what is its role? goals? And in the national fiscal year budget 19. Are there any sector ministries that you allocation decisions? would suggest represent a good model for 28. How would you describe the fiscal year using performance-based information to budget monitoring that the Ministry of manage the activities and programs? Finance or Ministry of Planning do of the 20. Are there any public sector reforms (with or sector or line ministries--none, light, without donor support) that are taking medium, heavy? Can you give some ex- place in the national government that in- amples to support your choice? clude efforts to strengthen systems to collect 29. Is there any evidence that donor reporting and manage information related to govern- requirements either conflict with one an- ment performance? other or impose duplication for the govern- 21. How would you assess the government's ment in meeting these requirements? I-PRSP and full PRSP (as well as CDF, if 30. What kind of financial expenditure data are relevant) documents in terms of their inclu- collected--and by whom--on the costs and Annex I 177 outputs of the functions and activities of the within the central and sector or line min- national government? istries have available to them? 31. Can you describe what financial expendi- -budget data ture data are collected--and by whom--on -output data the costs and outputs of the functions and -outcome or impact data activities of regional or local governments? -performance audits 32. How available are expenditure data to per- -financial audits sons and organizations outside the govern- -project and program completion reports ment? To civil society, to the media, to -donor data systems NGOs, to others? -other 33. Who in the government is responsible for the collection of socioeconomic and poverty Part III: Capacity Building Requirements for a data for the country? With whom are these Performance-based M&E System data shared? 38. How would you assess the skills of civil 34. What are the roles and responsibilities of servants in the national government in each the national statistics office? of the following six areas: -In what areas are statistics collected? -project and program management -At what levels in the country (city, re- -data analysis gional, national)? -policy analysis -To whom are the statistical data provided? -setting project and program goals -What information is or is not made public? -budget management -What organizations assist in collecting -performance auditing statistical information? 39. Are you aware of any technical assistance, -What special surveys are conducted, for ex- capacity building, or training in M&E now ample, Household Income and Expenditure underway or done in the past two years for Survey (HIES), HIV/AIDS, and others? any level of government (national, regional, 35. What are the roles and responsibilities of or local)? Please describe who provided this the National Audit Office? help. Has it been related to: -What is its authority to audit central and ­the CDF or PRSP process sector or line ministries? ­strengthening of budget systems -Does it have authority at regional and local ­strengthening of the public sector adminis- levels of government? tration -To whom are findings reported? -government decentralization -Are these findings made public? -civil service reform -Does the National Audit Office have any -individual central or line ministry reform? oversight on the quality of information 40. Are you aware of any institutes, research produced in the government? centers, private organizations, or universi- 36. Are there any organizational units in the ties in the country that have some capacity national government that have evaluation to provide technical assistance and training expertise and undertake evaluations? for civil servants and others in performance- 37. What data systems do the planning units based M&E? Annex II Readiness Assessment Toward Results-Based Monitoring and Evaluation in Egypt A World Bank Diagnostic Mission lending and encouraging greater transparency June 1­9, 2001 and accountability for results on the part of its Cairo, Egypt borrowers. Investments within the country-led Comprehensive Development Framework and Contents Poverty Reduction Strategy Papers depend upon tracking results (or the outcomes of govern- Executive Summary ment), rather than traditional monitoring and Background evaluation approaches, which typically track in- The International Experience puts and processes. Results-based monitoring Methodology and evaluation focus management on perform- Summary of Findings ance and on progress towards these desired de- Illustrative Current Technical Assistance velopment outcomes. Thus, this program also Activities supports the Bank's strategy of encouraging Moving to Results-Based Monitoring and countries to monitor progress on international Evaluation: Recommendations development goals. Near-Term Activities to be Supported by The Government of Egypt, through the Min- The World Bank ister of Finance, has expressed a desire to partic- Annexes ipate in the program. The Minister is eager to reform the budget to achieve a greater focus on Executive Summary improving the government's performance both in efficiency and effectiveness measures. The Background. In September 2000, the Board of Minister and others in the Government of Egypt Directors of the World Bank approved a pro- understand that a new focus on results is both gram to strengthen results-based monitoring necessary and consistent with the many efforts and evaluation in the operations of the Bank underway to reform public management systems and its borrowers. Both borrowers and the the world over. Bank need good information on performance to allocate resources wisely, design and implement Methodology. For those countries participating projects and programs effectively and evaluate in the Monitoring and Evaluation Improvement the effects of their activities on the achievement Program, the first action undertaken by the of development goals. World Bank is to conduct a short diagnostic For the World Bank, this program, called the study in order to evaluate the status of results- Monitoring and Evaluation Improvement Pro- based monitoring and evaluation in that coun- gram, is particularly important at a time when try and to identify opportunities for strengthen- the Bank is shifting toward more programmatic ing performance-based efforts both underway 178 Annex II 179 and planned. The World Bank, (through its thereof) contributed to the use of monitoring Operational Policy and Country Services and evaluation in the country context? Organization) conducted a diagnostic mission Areas Recommended for Moving Forward to Egypt on June 1-9, 2001 (see Annex A). The diagnostic team met with many key gov- First, Establish cross-ministerial leadership ernment officials, academics, donors and others group to promote performance and results- outside the government and reviewed a variety based monitoring and evaluation. A leadership of reports and documents to learn how a shift to team of ministers who are committed to change results based monitoring and evaluation could in their own organizations could accelerate the strengthen effective public management in Egypt adoption of results-based monitoring and evalu- (see Annex B). The team looked for organiza- ation and introduction of a more results-based tions and parts of organizations that are begin- budget process. Such a group, under the leader- ning to move toward results-based monitoring ship of the Minister of Finance, could play sev- and evaluation in order to achieve development eral key roles, for example: developing an over- goals. The team mapped monitoring and evalua- all strategy to guide the effort; providing guid- tion efforts currently underway and did an as- ance and an evaluation framework for pilot sessment of research and data collection capac- activity in individual ministries and other ity inside and outside the government. With an organizations; and developing a plan to expand eye to finding opportunities for strengthening pilot activity and share best practices and les- monitoring and evaluation, the team looked for sons across ministries. This group should deter- evidence of performance-based budgeting and mine whether mechanisms that other countries for innovation in these areas. At the request of have used to give impetus and mandates to H.E. the Minister of Finance, the team sought to reform efforts--such as presidential decrees, identify practical steps to encourage the devel- amendments to budget laws, and legislation-- opment of a "climate of performance" in the should be pursued in the Egyptian context. Egyptian government. The team's considerations included: Second, Support the initiative of the National · What is driving the need for results-based Council for Women to monitor and evaluate the monitoring and evaluation systems in the implementation of gender-related initiatives in Egypt (incentives/demands)? the 2002-2006 plan. Under the patronage of the · Where in the government does accountability First Lady, the Council has worked very effec- for effective (and efficient) delivery of pro- tively with the Ministry of Planning and line grams lie? ministries as they have developed their plans. · Is there a codified (through statute or man- We believe that the next step of monitoring and date) strategy or organization in the govern- evaluating implementation presents a particular ment for tracking development goals? opportunity to be a catalyst for change across · Where does capacity lie with the requisite several ministries. Because the Council includes skills for designing and using results-based a broad range of actors from inside and outside monitoring and evaluation systems in the government, including academics, the private pilot country? How has this capacity (or lack- sector, non-governmental organizations, the 180 Annex II media, and concerned ministries, it can help tics to encourage their widespread dissemina- promote consensus about measurement issues tion, within the government and to non-govern- and transparency in reporting on results. mental users. To better understand the needs of their clients, agencies responsible for producing Third, Build capacity to support reform. No statistics could create advisory groups to repre- country has succeeded with a significant reform sent users. Another useful function of such effort without a dedicated, well-organized team groups would be to encourage the exchange of to support it. A core team could support the information between data users, who may find ministerial group and the pilots so as to mini- solutions to common problems. Such advisory mize bureaucratic red tape and expedite innova- groups would meet regularly with the managers tion and learning; identify lessons learned; and of statistical units in the agencies. At the highest determine ways to mainstream these lessons level, an advisory group to CAPMAS or the into the government. The team could draw on proposed statistical commission would provide the career staff of several ministries and upon input on the needs of all users of Egyptian sta- the significant resources in the Egyptian aca- tistical information. demic and non-governmental community. Fourth, Modernize statistical policy. Despite Sixth, Participate in the IMF Special Data the manifest capacity of the Egyptian statistical Dissemination System. As Egypt prepares to system, there is evidence that it lags behind enter international capital markets, it will both in the quality of statistics produced and become more important for it to produce the attention given to the needs of its clients. credible statistics. An important step in this Egypt should review its statistical law with a direction would be subscription to the Special view to separating the responsibilities of the Data Dissemination Standard. (See the IMF Central Agency for Public Mobilization and disseminations standard bulletin board: Statistics (CAPMAS) for military mobilization http://dsbb.imf.org.) The SDDS requires coun- from its role as an independent statistical tries to adopt international standards for agency. Many countries employ a national sta- reporting on major economic and financial sta- tistical commission or similar coordinating tistics and to maintain a current listing of its body to set policies and standards for the pro- policies and standards. Working toward SDDS duction and dissemination of official statistics. participation would provide a powerful driver Egypt should adopt a similar strategy for coor- for modernizing Egypt's statistical system. dinating data policies and standards. Such a commission should include in its membership Finally, Donor support. There is an important both representatives of the agencies charged role for donors to play in supporting Egypt's with producing statistics and senior statisticians shift to results-based monitoring and evaluation drawn from universities and the private sector. with training and technical assistance. In doing so, donors can draw both on in-country expert- Fifth, Increase client focus. The value of statis- ise in universities and think tanks as well as the tics lies not in their production, but in their use. substantial international experience in results- It should be the goal of all producers of statis- based approaches. Annex II 181 Background underway to reform public management systems the world over. In September 2000, the Board of Directors of the World Bank approved a program to The International Experience strengthen results-based monitoring and evalua- tion in the operations of the Bank and its bor- For the past two decades, governments in devel- rowers. Both borrowers and the Bank need oped countries and, more recently, developing good information on performance to allocate countries have been "in search of results." Ac- resources wisely, design and implement projects cording to a recent OECD review," Improved and programs effectively and evaluate the effects performance of the public sector is a central fac- of their activities on the achievement of devel- tor in maintaining welfare of individuals and the opment goals. competitiveness of the economy. Performance For the World Bank, this program, called the management is the key aspect of public sector Monitoring and Evaluation Improvement Pro- reforms of many OECD Member countries." gram, is particularly important at a time when The strategies used to achieve greater per- the Bank is shifting toward more programmatic formance vary across countries, however, there lending and encouraging greater transparency appears to be a number of similar elements that and accountability for results on the part of its contribute to a successful shift to a results-based borrowers. Investments within the country-led culture. Among these elements are: Comprehensive Development Framework and · A clear mandate for making such a shift; Poverty Reduction Strategy Papers depend upon · The presence of strong leadership, usually tracking results (or the outcomes of govern- through a strong champion or champions at ment), rather than traditional monitoring and the most senior level of government; evaluation approaches, which typically track in- · The use of reliable information for policy and puts and processes. Results-based monitoring management decisions; and evaluation focus management on perform- · Economic pressures and other incentives for ance and on progress towards these desired de- change (often, a concerned citizenry or the velopment outcomes. Thus, this program also need to reduce the cost of burdensome civil supports the Bank's strategy of encouraging service payrolls); countries to monitor progress on the interna- · Clear links to budget and other resource allo- tional development goals. cation decisions; The Government of Egypt, through the Min- · Involvement of civil society as an important ister of Finance, has expressed a desire to partic- partner with government; and ipate in the program. The Minister is eager to · Pockets of innovation that can serve as begin- reform the budget to achieve a greater focus on ning practices or pilot programs. improving the government's performance both There appears to be no one right way to in- in efficiency and effectiveness measures. The troduce performance management into the Minister and others in the Government of Egypt many institutions and policy-making activities understand that a new focus on results is both of government. Often, depending on the pres- necessary and consistent with the many efforts ence (or absence) of the elements listed above, 182 Annex II governments try one or more of the following plement a more comprehensive strategy. Still strategies: 1) comprehensive or whole-of- other countries have found it useful to focus on government approach, 2) sector specific, or the customers or beneficiaries of government 3) customer focused. services or on one client group, such as In the comprehensive approach, a number of women/girls or children. This strategy includes countries have introduced strategic plans, per- developing key performance indicators within formance indicators, and annual performance line ministries with a specific focus on improv- plans over a period of years and integrated then ing those government programs to support a into annual budget documents (Australia, particular group of citizens. This strategy can United States). Other approaches include put- also help to move forward a national agenda ting program performance indicators in the in a program area, rather than waiting for the annual financial reports that can be audited entire government to embrace performance (Finland, Sweden, United States) or using per- management. formance agreements between ministers and Other strategies used by governments to in- heads of government agencies (New Zealand, troduce performance management include: United Kingdom). Argentina and Romania are · Selection of free-standing authorities, grant- also piloting performance-based budgeting ing them greater flexibility to use resources strategies. Here, performance indicators for and hold their leaders responsible for results. government programs are linked to allocated Examples: Next Steps Agencies, Performance budget envelopes; reported in budget annexes Based Organizations. at the start of each budgeted year; and audited · Encouragement and recognition of pilot ac- at year's end. And some countries, such as tivities within many organizations that can Malaysia, have embraced the total quality man- lead the way and be replicated in other agement approach, focusing on process reengi- places. Example: Reinvention Laboratories. neering and achieving strict quality standards. · Introduction of total quality management: While most of the OECD countries have this model, developed by industry to improve adopted a whole-of-government approach to in- manufacturing processes has been applied in troduce performance management, many coun- a few countries to public sector reform, gen- tries, like the United States, began with perform- erally after the reform process is well under- ance pilots. By first piloting in a few programs way. The focus of quality management on and sectors, governments hoped to create favor- customer requirements is relevant to reform able conditions for public sector learning and efforts at all stages. Example: Malaysia. experimentation before " mainstreaming" the No strategy can simply be mapped from one effort. Other countries find that moving forward country or situation to another. Furthermore, in in those sectors where a clear reform effort is practice, the strategy that is used by a given underway (for example, the health sector in country at a particular point in time may be a Bangladesh, Ghana and the Kyrgyz Republic) combination of one or more approaches like allows innovative efforts to move forward, re- these. Furthermore, reform efforts are multi- gardless of whether commitments have been year affairs and strategies inevitably evolve made by the president or prime minister to im- over time. Annex II 183 Methodology · Where does capacity lie with the requisite skills for designing and using results-based For those countries participating in the Moni- monitoring and evaluation systems in the toring and Evaluation Improvement Program, pilot country? How has this capacity (or lack- the first action undertaken by the World Bank is there-of) contributed to the use of monitoring to conduct a short diagnostic study in order to and evaluation in the country context? evaluate the status of results-based monitoring and evaluation in that country and to identify Summary of Findings opportunities for strengthening performance- The team found significant interest in moving based efforts both underway and planned. The toward a "climate of performance." This was World Bank, (through its Operational Policy described in various ways in our interviews, and Country Services Organization - OPCS) but the interviewees seemed to have in common conducted a diagnostic mission to Egypt on June a desire to use good information to allocate re- 1­9, 2001 (see Terms of Reference for this mis- sources, assess progress, and achieve develop- sion in Annex A). ment goals in the most effective and efficient The team looked for organizations and parts way. Outlined below are the major findings of organizations that are beginning to move to- from the diagnostic assessment. In each section ward results-based monitoring and evaluation in the team will present both opportunities noted order to achieve development goals. The team and potential obstacles for shifting the focus of mapped monitoring and evaluation efforts cur- government in Egypt to achieving results. rently underway and did an assessment of re- search and data collection capacity inside and outside the government. With an eye to finding Leadership opportunities for strengthening monitoring and Successful efforts to shift the focus of govern- evaluation, the team looked for evidence of per- ment to results have enjoyed high levels of formance-based budgeting and for innovation in sustained leadership. Successful reforms have these areas. At the request of H.E. the Minister generally been led from the executive branch ­ of Finance, the team sought to identify practical from the Cabinet Office (United Kingdom), the steps to encourage the development of a "climate Treasury (New Zealand), the Vice President of performance" in the Egyptian government. (United States), or the Chief Minister (Andhra The team's considerations included: Pradesh, India). · What is driving the need for results-based In Egypt, the team noted the interest ex- monitoring and evaluation systems in the pressed in shifting to a climate of performance Egypt (incentives/demands)? on the part of many senior government officials, · Where in the government does accountability including the Prime Minister and the Cabinet. for effective (and efficient) delivery of pro- The President himself has called for better infor- grams lie? mation to support economic decision-making. · Is there a codified (through statute or man- The First Lady chairs the National Council for date) strategy or organization in the govern- Women, which is developing a system to moni- ment for tracking development goals? tor and evaluate efforts across many ministries 184 Annex II to enhance the status and condition of women (New Zealand), key legislation (such as the in Egypt. Government Performance and Results Act in The Minister of Finance is playing a key lead- the United States), allowing flexibilities and in- ership role. He has a strong desire to reform the centives, or conducting studies, audits or hear- Egyptian budget to better support performance. ings on government performance. This aspect In meetings with the diagnostic team, he under- was beyond the scope of the team's exploration scored the importance he places on giving in- at this time; it may be useful to address the role creased attention to improving the management of the Egyptian legislature, the People's Assem- of public expenditures, noting that "I want to be bly, the Shura Council and the Central Audit able to tell Egyptian taxpayers that the govern- Organization, which reports to them, in the ment is spending their money efficiently and future. effectively." Incentives or Key Drivers For an effort to be successful, it is also impor- tant that the line ministries--who are responsi- In most countries that have moved to a results- ble for resource expenditures and overseeing the based system, there has usually been a clear implementation of specific programs--be fully driver for reform. For some, entry into the Euro- engaged. The team found significant interest in pean Union provides an incentive for change. In monitoring and evaluating for results on the countries seeking debt relief, change has been part of several line ministers. The Minister of driven by a requirement to develop a Poverty Electricity, who led reform efforts at the Interna- Reduction Strategy Paper, which includes a well- tional Atomic Energy Agency before assuming constructed performance-based monitoring and his current responsibilities, recommended that a evaluation framework of indicators and per- group of ministers concerned with improving formance measures. For some developed and de- the management of critical infrastructure and veloping countries, significant deficits have public utilities take on a leadership role. brought cuts in government spending and forced A recent review of the Egyptian civil service a greater focus on government efficiency and ef- underscores the importance of the leadership of fective allocation of resources. For others, public such ministers in a few countries, "The most dissatisfaction with the cost and performance of important point is the interest and commitment government has become a political issue, result- shown by the head of the organization--the ing in political commitments that have driven minister or the senior-most bureaucrat in the change. organization . . . the ministers of Egypt enjoy The team did not find any single compelling considerable degrees of freedom and influence in driver of change in Egypt. During the second their respective ministries to introduce changes half of the 1990s, economic growth has been in organization and management of the person- robust. Although the deficit reached up to 3 nel. `Hands-on-management' concept is really percent of GDP in 1998 and there are other practiced by strong executives in Egyptian qualitative weaknesses in economic perform- government"1 ance, economic drivers are not sufficient to Finally, in many countries, the legislative arm create a compelling need for change. Rather of government has also played an important than a single driver, however, the people we in- leadership role, by enacting a reform framework terviewed suggested a variety of reasons that Annex II 185 are driving different actors in the public sector For example, Presidential Decree No. 90 estab- to give greater consideration to performance: lished the National Council for Women in Feb- · Egyptian-European Partnership Agreement. ruary 2000 and directs the Council to "follow The Prime Minister has recently stressed the up on and evaluate the implementation of public importance of completing plans to modernize policy on women's issues" in addition to advis- the State in conjunction with the signing of ing on policy and other responsibilities. This the Egyptian-European Partnership Agree- council, chaired by the First Lady is composed ment. This agreement is also the reason for of thirty members from government, academia, the urgency of an industrial modernization media and other organizations. It is now work- program to prepare Egyptian industries to ing with the Ministry of Planning and the line compete with foreign products; ministries as they are preparing Egypt's next · Presidential decree corporatizing economic five-year plan to assure that the issues that most authorities. The economic authorities such as affect women are reflected in that document. the Rail Road Authority , the Cairo Water Au- The Council intends to put in place a system to thority and the Electricity Authority currently monitor and evaluate the implementation of the receive LE 3 billion in annual subsidies and plan to fulfill the mandate specified in the Presi- more than LE 280 billion in cumulative invest- dential Decree. ments. The government aims to improve the The Information and Decision Support Cen- performance of these authorities and move ter of the Egyptian Cabinet has the mandate to them towards a privatization strategy; and establish and operate information centers in all · Donor interest. Several donors have an ex- of the governorates of Egypt to support deci- plicit interest in enhanced performance of the sion-making. An additional example is the Cen- public sector and are providing related train- tral Audit Organization (CAO) with a long- ing, technology and technical support (see standing legal mandate to conduct audits of below). The World Bank has identified the performance as well as financial auditing. In this creation of a more results-based budget case, however, the people we spoke with re- process as one of its priorities. ported that CAO is almost exclusively focused on financial issues. Mandates or Clear Authorities In summary, the team did not identify any Countries that have embarked on a significant over-arching mandate in Egypt that would guide program to shift to a results focus have not only a substantial shift to results-based monitoring had a reason to change, they have generally es- and evaluation. Moreover, the team did not tablished a formal mandate to do so. This has identify existing legislation or decrees that pro- taken a variety of forms, for example legislation, vide the framework and authority for broad presidential or prime ministerial decrees, or ex- change. ecutive orders. In some cases, countries have A Well-Defined Strategy found that sufficient authority exists but that existing mandates have not been fully imple- A key criterion for a successful shift towards re- mented. sults requires the development of a well-commu- In Egypt, there are a number of individual or- nicated and executable strategy. This strategy ganizations or groups with specific mandates. should be directed by senior government leader- 186 Annex II ship and embody a clear mandate for informed ernment while ignoring the millions of civil ser- decision-making and a focus on achieving de- vants who do the work and interact with citi- sired results of government. Supporting the zens. A successful strategy should be responsive strategy should be an implementation plan that to the real needs of citizens as they interact with includes clear goals, direction and timelines. their government. And it should include fre- Recognizing that a shift to a performance ori- quent monitoring and adjustment to keep it on entation will require a significant multi-year track. A successful strategy should support the effort, Egypt's Minister of Finance has begun to leaders of change--giving them sufficient train- define an approach with several aspects. He ing and coaching, allowing them to take risks wants to draw on external examples of reform and be recognized for successes. Finally, of and recently supported the visit of a team of course, a successful strategy is one that is turned government officials to Malaysia to look at its into actions, brings about real changes in the experience. He has identified a few activities un- performance of government, and increases the derway that could become pilot tests for per- efficient and effective use of resources formance budgeting. He recognizes the impor- Pockets of Innovation tance of using good data for monitoring and evaluating the effort. This is an excellent start- The team found a number of pockets of innova- ing point for developing an effective strategy. tion in performance measurement in Egypt The next critical element is to develop a showing that it is feasible to shift to a results- broader strategy for change that is both effective based approach in the Egyptian context and and bold, yet practical and feasible. Such a providing useful starting points for a broader strategy should provide a framework for various effort. Several ministries have ongoing pilot ongoing efforts to shift to a results-based system activities or centers of excellence that include a for the use of public expenditures as well as greater emphasis on results. The team heard serve to stimulate and guide additional initia- about initiatives in the Ministry of Electricity tives. There is no single answer for what consti- and Energy and in the Petroleum Authority as tutes the best strategy. Rather the leadership well as the Broadcasting Authority. The team team should develop a strategy that reflects the met extensively with the Minister of Health and constraints and opportunities in Egypt at this Population, to discuss his strategy for collecting time and start the process of change. real-time health statistics to support better In developing its strategy, Egypt may wish to health policy-making. consider a number of lessons from international The Ministry of Finance itself has identified experience while building on its own consider- several pilot activities that are introducing new able experience and expertise. First and fore- approaches to management. For example, the most, a successful strategy will need to be clear National Center for Education Research and about its objectives. It must be simple and easy Development has introduced a number of inno- to communicate. It should engage and inspire vations in a program to reduce educational dis- government employees to help bring about parities and enhance quality of education. With change as it will not be possible to increase the support from the European Union, the World efficiency and effectiveness of the Egyptian gov- Bank and the Government of Egypt, the Center Annex II 187 is working with fifteen governorates, mapping has launched one of Egypt's first e-government areas with the greatest need and consulting local projects that will allow citizens to do govern- communities on their priorities. The Ministry of ment transactions online and at kiosks. Finance is also adopting a results focus in its Debt Management Unit and in the Sales Tax At the same time, we found that these efforts Unit where it is seeking to introduce a program to shift to results-based management remain of total quality management. fragmented and disconnected (further discussion One of the most innovative approaches we of this issue is found below). There is little in- saw was the assessment system of the Social centive or opportunity to share information or Fund for Development, an Egyptian government lessons across organizational boundaries. organization that administers funds from some Information Driving Decision-Making seventeen donors for development projects. To help guide their allocation of resources, they During the team's visit to the National Center have developed a Community Needs Assessment for Educational Research and Development, we application, which includes a set of composite saw a clear example of a key decision-maker indicators for health status, education, infra- using information to make policy. During the structure and housing conditions, and other meeting with the team, the Minister of Educa- basic needs. Each index combines information tion called to ask the Center's director for the on service levels and the impact on peoples' well results of a review of international practice on being. The resulting indexes, disaggregated to the frequency of student testing. He had asked the district level, will be combined with informa- her to carry out the review as input to his con- tion on population size and restrictions imposed sideration of changes to Egypt's practice of an- by donors, to allocate Social Fund resources. The nual testing. This real-time example of a senior allocation system is currently undergoing sensi- policy-maker seeking out good information to tivity testing before being deployed. support decisions is the hallmark of a modern results-based monitoring and evaluation system. The Cabinet's Information and Decision Through interviews we learned of other inno- Support Center is providing technical support vative applications of research and statistical in- for decision-making to Egypt's Cabinet and to dicators for improving the quality of decision- its governors and district officials through a na- making in the Egyptian government. Many of tional network of information centers. The the International Development Goals and indi- Center has integrated national databases and is cators to measure the goals are incorporated in now making them available to local govern- data sets used by Egyptian agencies to monitor ment officials via CD-ROMs and to the public their programs (see Annex C). For example, the via the Internet. The Center also produces Ministry of Health routinely tracks a set of per- monthly economic bulletins and an annual sta- formance indicators to monitor the success of its tistical report on the nation and on each gover- overall program. However, according to our in- norate. It assists local government entities to terviews, such examples of using research and produce annual statistical reports and to de- statistical information as inputs to decision- velop and maintain local websites. The Center making are still the exception rather than the 188 Annex II rule. One particularly pessimistic review of the cil and a program for journalists on covering the research environment in Egypt noted, "The re- public budget. search outputs are not usually considered by the Statistical System. Egypt's statistical system has policymakers, no matter how relevant the re- a long history, and has grown into a large and search topics to the problems that the country is multifaceted system, capable of carrying out facing, or how sound the analyses and conclu- complex studies on a large scale. The value of sions reached. In Egypt, the design and applica- high-quality statistical information is recognized tion of policies are neither supported nor guided inside and outside the government. The expan- by serious research."2 sion of information communication technology throughout Egypt has increased the capacity of Research Capacity. Egypt has significant re- the statistical system and resulted in innovative search and statistical capacity and well-trained efforts to use statistics for planning, monitoring, researchers in both public and private institu- evaluating, and decision support. As Egypt tions, which are a significant resource for deci- moves to implement performance-based man- sion-makers seeking to shift to a greater focus agement techniques, one of its strengths is its on results. One of the private centers, Egypt's statistical system. At the same time, there are Economic Research Forum, is the regional hub weaknesses in the system that must be for the World Bank-sponsored Global Develop- addressed if Egypt is to move forward rapidly ment Network. It is being considered as a can- and with confidence in the quality of its official didate to become an International Center of statistics. Excellence. These Centers will be part of the "Evaluation Partnership Program for Monitor- Sources of Official Statistics. The principle ing and Evaluation Capacity Development" sources of official statistics in Egypt are the that is in place between the Policy and Opera- Central Agency for Public Mobilization and tions Evaluation Department of the Ministry Statistics (CAPMAS), the Ministry of Planning, of Foreign Affairs (Government of the Nether- the line ministries (including Economy, Finance, lands) and the World Bank's Operations Evalu- Health, and Education), and the Central Bank. ation Department. In addition numerous studies that have pro- duced specialized databases have been carried The Social Research Center at American Uni- out by academic research centers and non- versity of Cairo has developed a program of governmental organizations, often in collabora- training courses for evaluators and its director is tion with government agencies. CAPMAS has leading the monitoring and evaluation effort two roles in the statistical system: it collects and with the National Council for Women. In addi- disseminates statistics and it is the authorizing tion, the Public Administration Research and agency for all statistical research carried out in Consultation Center at Cairo University has a Egypt. In the latter capacity, CAPMAS reviews program of research and training on public ad- all proposed survey instruments; it may require ministration including recent programs on lead- changes to or deletion of items on the instru- ership for the top management of Egypt's Elec- ment; and it receives a copy of all data collected tricity Authority, training on decision support through authorized instruments. In addition, for Egypt's People's Assembly and Shura Coun- CAPMAS' role as the agency for public mobi- Annex II 189 lization has disposed it to a very restrictive view budgetary items) and are slow in closing; of what statistics may be published, viewing national account statistics (now produced by many official statistics as having military value. the Ministry of Planning with input from Likewise it has been very sensitive to the types CAPMAS) are not compiled to current stand- of questions asked by private researchers. ards, are not complete, and have been arbitrari- However, in the past five years, we were told, ly revised. In both cases, technical assistance CAPMAS has adopted a more liberal standard projects supported by donors are underway and for what data can be disseminated and routine- should result in substantial improvements. In ly authorizes questionnaires. other cases, the doubts concerned the appropri- ate definitions of and methodologies for calcu- Expansion of Statistical Activities. Although lating poverty, illiteracy, and unemployment Egypt appears to have a highly centralized sta- rates. The lack of faith in the quality of statis- tistical system, we learned during our mission tics has a corrosive effect on public dialogue: that many agencies are developing their own debates over how to address the serious issues data systems, which in some cases go far beyond of development devolve into conflicting claims the traditional collection of administrative sta- about the accuracy of the statistics. tistics. For example, because the Ministry of Health and Population regards health as a com- The Role of CAPMAS. Concerns were also raised prehensive concept involving both physical and about the role of CAPMAS. Academic researchers social well being, it is developing a program for and others outside the government felt that its collection of health and social statistics at the role as the authorizing agency for survey household level. This complements its manage- research exerts a deadening influence on inde- ment information system which, when com- pendent studies. CAPMAS' capacities, especially plete, will integrate health records from over for executing surveys, is widely recognized, but 4500 primary care units into a national data- it is not viewed as an innovator or a source of base. Others are proceeding to develop new sta- leadership in advancing statistics in Egypt. tistical measures, which are not available from Despite the liberalization of dissemination rules, CAPMAS or other sources. The Ministry of the CAPMAS appears to see its principal role to be Economy, for example, has begun work on pro- the regulation, rather than the creation, of duction indexes and a set of "leading indica- information. Despite its leading role as the tors" for the economy. The expansion of statis- national statistical office of Egypt, CAPMAS tical activities beyond the traditional collection does not participate in international statistical and reporting systems raises new challenges for forums, such as the United Nations Statistical maintaining standards and ensuring comparabil- Commission, nor has it expressed interest in ity across different data sets. joining the IMF's General Data Dissemination System or working toward the Special Data Quality of Statistics. During our interviews, we Dissemination Standard. heard many concerns raised about the reliability of statistics in Egypt. Some of these concerns Measuring Organizational Performance. What were based on well-documented inadequacies: the team did not find was any systematic collec- the fiscal statistics from the Ministry of Finance tion and use of data to measures client satisfac- are not complete (because of numerous extra tion. While these are not development "results," 190 Annex II they are important measures of organizational conducive to a performance orientation. The performance. Experience has shown the impor- Egypt Social and Structural Review identifies tance of a balanced scorecard of results that several issues that suggest the magnitude of the includes these aspects alongside the major socie- challenge that will be required to move to a tal and financial outcomes that are expected. budget process that is focused on performance: · Prioritization of Expenditures. The current Links to Resource Decisions budget process does not include common ap- The budget is a key instrument in any country proaches to encouraging prioritization such for making choices about priorities and imple- as providing budget ceilings or envelopes to menting governmental policy. Recently, as an encourage line ministries to prioritize budget OECD official noted, "There has been a quiet requests. In addition, ad hoc budget negotia- revolution in the methods and philosophies of tions and revisions during the year further un- budgeting that began in a few developed coun- dermine the implementation of budget priori- tries in the 1980s, and it is being felt around the ties established by the Cabinet and the world. Most countries have embarked on some People's Assembly. sort of reform to budgeting aimed at improve- · Incentives for Efficient Service Delivery. The ments in macroeconomic stability, improved pri- budget process does not reward efficient serv- oritization of expenditure and more effective ice delivery either in budget negotiations or policy implementation." through incentives such as sharing of savings Egypt's budget process does not currently or allowing greater flexibility in how re- lend itself to prioritizing expenditures or effec- sources may be used. tively assuring implementation of policy. Fo- · Transparency. Information about the Egypt- cused on finances and other inputs, neither the ian budget is restricted to a high degree. The budget process nor its format facilitate linking budget approved by the People's Assembly is funds with their intended result. In addition, the not made public; sections of budget docu- budget approval process used by the People's ments are made available on a "need to Assembly, the controls exercised by the Ministry know" basis, basic financial statistics are not of Finance and the oversight of the Central Audit- published or published in a very aggregated ing Organization are focused on financial aspects form and audit reports are narrowly dissemi- without regard to their relation to outcomes. nated and do not include information on the The World Bank has identified the budget as effectiveness and efficiency of expenditures. a priority area for reform in Egypt, noting that · Comprehensiveness. Responsibility for the "despite robust economic growth, social out- preparation and execution of the Egyptian comes--especially in health and education-- budget is divided between the Ministries of have not improved at a commensurate rate . . . Finance and Plan (see table). This makes re- The first step to ensuring that the nation's re- sponsible fiscal policy and realistic planning sources are better spent in these areas is by tak- difficult since investment projects can have a ing steps to improve the results orientation of large impact on overall budget levels and the budget, especially the recurrent budget."3 make it difficult to project recurrent cost The current budget structure and process is not requirements. Annex II 191 Government of Egypt: Budget Categories Ministry of Finance Ministry of Plan Chapter 1 Chapter 2 Chapter 3 Chapter 4 Wages & salaries (including Materials & Investment Debt service allowances) supplies expenditures payments Implementing a Workable Strategy team" ­ bringing together people responsible for writing the national plan in all of the ministries. There is strong interest on the part of several The small core group at the Council held work- ministers in coming together to shape an effort shops for these cross-ministerial groups to orient to strengthen the use of results-based monitor- them to the Council's concerns and has subse- ing and evaluation in the government. They may quently worked with them as they wrote their consider forming a leadership team that: individual ministry's plans. · Defines objectives and develops a strategic The Minister of Finance expressed strong vision; interest in training activities to support this ini- · Provides a timeframe and evaluation frame- tiative. There are few specific courses that are work for pilots and other activitiesl focused on performance budgeting and results- · Measures progress; and based monitoring and evaluation. The World · Recognizes and supports progress. Bank, through its OPCS and OED units, has It will be important to support the leadership developed a course on Developing and Building team with a dedicated, well-organized group to Performance-Based-Monitoring and Evaluation assure that the vision gets turned into action. Systems for Government Officials. There is also The roles that such a group might play include the International Program for Development developing an action plan, assuring that there is Evaluation Training to be held in Ottawa, adequate training and support is in place, meas- Canada in July (the launching of this program uring progress, identifying successful efforts and is July, 2001). There are also substantial re- sharing lessons learned across organizational sources both in Egypt and internationally boundaries. The group should include commit- that could be drawn upon for training. As ex- ted, energetic individuals, with no other job to amples, the Institute of National Planning has attend to. The group should have a well-com- a course on performance budgeting that was municated and clear authority from the minister, developed for the Cairo Water Authority; the prime minister, or president. Social Research Center has a training program In developing such a team, it is probably de- on monitoring and evaluation that could be sirable to draw upon more than one ministry built upon; and the Public Administration and to assign people to a cross-ministry team for Research and Consultation Center at Cairo Uni- time-limited assignments. The National Council versity has developed several relevant training for Women developed a very interesting "virtual programs. 192 Annex II Donor Sponsored Activities tate access to information in the legislative process. According to project documents, During the diagnostic mission, the team met there has been an increased demand by mem- with USAID, the largest single donor in Egypt. bers of the Peoples' Assembly for better infor- While USAID does not have any specific plans mation, especially from government agencies, to provide additional technical assistance in the and for quantitative information on topics area of performance-based monitoring and eval- under debate. uation, USAID's extensive experience in this · The IMF has been providing short-term tech- area is a significant resource and USAID ex- nical assistance to help the authorities in ad- pressed an interest in working more closely with dressing the shortcomings in the national the World Bank in this area. database, mainly in the areas of national ac- There are a number of donors providing tech- counts, balance of payments, monetary, fiscal, nical assistance to the Egyptian Government and prices statistics. Long-term assistance had who can further Egypt's shift to a results-based been provided in the past in areas of balance focus. A few ongoing activities that are sup- of payments and external debt. ported by donors are listed below: · UNDP is providing support and technical as- sistance both to the Information and Decision Illustrative Current Technical Assistant Support Center and to the Ministry of Finance. Activities · The Data Access and Transmission Activity Moving to Results-Based Monitoring and (DATA) Project is assisting the Government Evaluation: Recommendations of Egypt to develop and maintain national ac- counts compliant with the 1993 System of Shifting a public sector institution to focus on National Accounts. This project, funded by performance, much less an entire government, USAID, is upgrading the information tech- requires a major, multi-year effort with strong nology systems in the Ministry of Planning leadership and commitment to change. Even and providing technical assistance to governments that embarked on this course more strengthen Egypt's collection, tabulation, and than a decade ago are still evolving. Interna- dissemination of key economic data. tional experience has also shown that sometimes · Egypt's Industrial Modernization Programme the biggest pay off is from some of the early, is assisting Egypt's industrial sector to prepare relatively straightforward steps on a much for trade liberalization. Funded by the Euro- longer journey to a culture of performance that pean Union, the Government of Egypt and effectively links resources and results. The Gov- the private sector, the project is providing ernment of Egypt is at the beginning of this technical assistance to small and medium en- journey. We believe the following steps will ad- terprises and to the sector overall. Currently a vance it. Danish team is reviewing Egypt's national sys- · Establish a cross-ministerial leadership group tem for quality. to promote performance and results-based · A USAID-funded activity has installed a com- monitoring and evaluation. A leadership team puter network in Egypt's Parliament to facili- of ministers who are committed to change in Annex II 193 their own organizations could accelerate the ministerial group and the pilots so as to mini- adoption of results-based monitoring and mize bureaucratic red tape and expedite inno- evaluation and introduction of a more results- vation and learning; identify lessons learned based budget process. Under the leadership of and ways to mainstream these lessons into the Minister of Finance, such a group could the government. The team could draw on play several key roles, for example: develop- the career staff of several ministries and upon ing an overall strategy to guide the effort; the significant resources in the Egyptian aca- providing guidance and an evaluation frame- demic and non-governmental community. work for pilot activity in individual ministries · Modernize statistical policy. Despite the mani- and other organizations; and developing a fest capacity of the Egyptian statistical sys- plan to expand pilot activity and share best tem, there is evidence that it lags behind both practices and lessons across ministries. This in the quality of statistics produced and the group should determine whether mechanisms attention given to the needs of its clients. that other countries have used to give impetus Egypt should review its statistical law with a and mandates to reform efforts--such as view to separating the responsibilities of presidential decrees, amendments to budget CAPMAS for military mobilization from its laws, and legislation--should be pursued in role as an independent statistical agency. the Egyptian context. Many countries employ a national statistical · Support the initiative of the National Council commission or similar coordinating body to for Women to monitor and evaluate the im- set policies and standards for the production plementation of gender-related initiatives in and dissemination of official statistics. Egypt the 2002-2006 plan. Under the patronage of should adopt a similar strategy for coordinat- the First Lady, the Council has worked very ing data policies and standards. Such a com- effectively with the Ministry of Planning and mission should include in its membership line ministries as they have developed their both representatives of the agencies charged plans. We believe that the next step of moni- with producing statistics and senior statisti- toring and evaluating implementation pres- cians drawn from universities and the private ents a particular opportunity to be a catalyst sector. for change across several ministries. Because · Increase client focus. The value of statistics the Council includes a broad range of actors comes not in their production, but in their from inside and outside government, includ- use. It should be the goal of all producers of ing academics, the private sector, non- statistics to encourage their widespread dis- governmental organizations, the media, and semination, within the government and to concerned ministries, it can help promote non-governmental users. To better understand consensus about measurement issues and the needs of their clients, agencies responsible transparency in reporting on results. for producing statistics could create advisory · Build capacity to support reform. No country groups to represent users. Another useful has succeeded with a significant reform effort function of such groups would be to encour- without a dedicated, well-organized team to age the exchange of information between support it. A core team could support the data users, who may find solutions to com- 194 Annex II mon problems. Such advisory groups would chartered inter-ministerial Group. The main meet regularly with the managers of statistical theme of this workshop will be performance- units in the agencies. At the highest level, an based budgeting, drawing on international expe- advisory group to CAPMAS or the proposed riences and resulting in the creation of a vision statistical commission would provide input and action plan for Egypt. Approach: interac- on the needs of all users of Egyptian statisti- tive. Duration: Two days. In the first day, World cal information. Bank experts will share their practical views · Participate in the IMF Special Data Dissemi- based on hands-on international experiences. nation System. As Egypt prepares to enter H.E. the Minister of Finance will introduce and international capital markets, it will become discuss the vision, strategy and action plan for more important for it to produce credible sta- Egypt in the second day. The timing of this ac- tistics. An important step in this direction tivity is expected to be early fall, 2001. would be subscription to the Special Data Suport the National Council for Women as it Dissemination Standard (see the IMF dissemi- prepares to develop a monitoring and evaluation nations standard bulletin board: http://dsbb. framework for measure the implementation of imf.org). The SDDS requires countries to the gender-related objetives established in all rele- adopt international standards for reporting vant ministries as part of the 2002­2006 national on major economic and financial statistics plan expected to be finalized in October 2001. and to maintain a current listing of its policies The National Council for women has specifi- and standards. Working toward SDDS par- cally requested technical assistance in the area ticipation would provide a powerful driver of designing and building a results-based moni- for modernizing Egypt's statistical system. toring and evaluation system to track the results · Encourage donor support. There is an impor- of gender-related programs im plemented across tant role for donors to play in supporting a number of line ministries. This support may Egypt's shift to results-based monitoring and include advice, consultation and training and evaluation with training and technical assis- should be linked to the upcoming World Bank­ tance. In doing so, donors can draw on in- sponsored gender assessment. country expertise in universities and think The World Bank should begin communicat- tanks as well as the substantial international ing immediately with the Council and its tech- experience in results-based approaches. nical advisors as to the timing and specifics of holding a consultation session in Cairo for key Near-Term Activities to be Supported by the members of the Council during early fall 2001. World Bank Curriculum for a possible trianing course will be developed in conjunction with the American 1. Provide technical support to the Minister of University in Cairo, and other research centers Finance in im plementing his vision to shift the with expertise in this area. The Secretary Gen- budget process to one that focuses on results. eral of the National Council for Women should Organize and coordinate a workshop or con- be a member of the inter-ministerial council sulting session aimed to directly support a newly discussed above. Annex II 195 2. Improve Egypt's statistical capacity. on the international capital markets. Although final acceptance of Egypt's metadata in the The goal of subscribing to the IMF Special Data SDDS may have to wait on completion of on- Dissemination Standard (SDDS) and assigning going work on its fiscal and national accounts, responsibility to the appropriate agency should the plan to subscribe, including a proposed date, be announced. Subscription to the SDDS will should be set as soon as possible. help Egypt in its plans to offer sovereign bonds Annexes Annex A. Terms of Reference Annex B. Interviews Conducted Annex C. Egypt and the International Development Goals Annex D. Endnotes, References, and Resources Annex A regularly monitored to assess progress in meet- ing development goals, and 2) a valid and verifi- Terms of Reference able system for data collection and reporting on Performance-Based M&E Diagnostic Mission those indicators. Egypt June 2­6, 2001 During the upcoming mission to Egypt, the mission team will meet with a number of officials Background in the Government, and in donor and other stake- Egypt is included among eight country pilots holder organizations to learn how performance- that have been selected to participate in a Bank- based M&E systems could support effective pub- wide program approved by the Board of Direc- lic management. The Team will begin to map tors in September. This Program, the M&E Im- M&E efforts currently underway and assess where provement Program, has as its main goal to help existing capacity in data collection and reporting both Bank and Borrower officials to better track lies inside and outside the government. The the results of development by strengthening Team will also assess where potential opportu- their use of performance-based monitoring and nities for designing and building performance- evaluation (M&E) systems. These systems (now based M&E systems and where potential barri- well-understood to support good public man- ers for being successful in this effort may exist. agement) can help government officials identify Meetings to be Held with Key Officials and set realistic goals and outcomes for public sector programs. Two key requirements of a us- Among those whose views will be important to able performance-based M&E system are 1) the understand will be officials from the following inclusion of performance indicators that will be organizations: 196 Annex II Ministry of Finance - Minister of Finance - Individuals involved in budget formulation - (If the proposed Fiscal Policy Decision Support Unit has been created, then the Team would like to meet with the head of this unit) - Individuals involved in the corporatization of the 62 public economic authorities Ministry of Planning - Individuals who prepare and oversee the investment budget - Head of the Statistical Office Prime Minister's Office Individuals who are directly responsible for setting sector priorities and overall economic development planning Central Accounting Office (CAO) Head of this office or senior officials in charge of ex post review of budget accounts Ministry of Health and Population; - Head of Administrative Reform Units Ministry of Agriculture and Ministry of - Head of Administrative data and Education reporting systems Ministry of Health and Population Head of unit that is responsible for outsourc- ing non-essential services Health Insurance Organization Director or lead individual Agriculture Research Center Official National Center for Educational Research Head Official National Statistical Office (or that office Head responsible for conducting household surveys) Donors One meeting with key donors, such as USAID, UNDP and others with a large presence in the country. Below is a partial list of questions that might be 2) Where in the government does accountability asked of these officials: for effective (and efficient) delivery of pro- 1) What is driving the need for results-based grams lie? monitoring and evaluation systems in the 3) Is there a codified (through statute or Egypt ? (incentives/demands) mandate) strategy or organization in the Annex II 197 government for tracking development the government with interest in designing a per- goals? formance-based M&E system to support sec- 4) Where does capacity lie with the requisite tor/program goals monitoring. Second, we hope skills for designing and using results-based to form partnerships with other key donors with monitoring and evaluation systems in the similar interest in helping Egypt build capacity pilot country? How has this capacity (or lack in the area of performance management for na- thereof) contributed to the use of tional and sector-wide programs. Finally, we M&E in country? plan to develop an action plan and set of recom- mendations that can be incorporated into a pro- Expected Outputs from the Mission gram aimed to strengthen the government or At the end of this mission, we hope to have key stakeholder's use of performance-based identified at least one or more champions within monitoring and evaluation. Annex B Interviews Conducted Economic Research Forum CAPMAS Information and Decision Support Center of Institute for National Planning the Egyptian Cabinet National Council for Women International Monetary Fund Public Administration Research Center, Ministry of Electricity & Energy Cairo University Ministry of Education Social Fund for Development Ministry of Finance Social Research Center Ministry of Health United Nations Development Programme Ministry of Industry and Technology United States Agency for International Ministry of International Cooperation Development Ministry of Planning Annex C Egypt and the International Development Goals Seven goals for international development have 1. Reduce the proportion of people living in been identified from the agreements and resolu- extreme poverty by half between 1990 and tions of the world conferences organized by the 2015; United Nations in the first half of the 1990s. 2. Enroll all children in primary school by These goals are: 2015; 198 Annex II 3. Make progress towards gender equality and dents) appear in the CAPMAS Statistical Year- empowering women by eliminating gender book or in the Human Development Report of disparities in primary and secondary educa- Egypt. Statistics on girls' enrollments are widely tion by 2005; reported, as are the proportion of women on 4. Reduce infant and child mortality rates by school and university faculties. Literacy rates two-thirds between 1990 and 2015; are also closely watched and reported on. Many 5. Reduce maternal mortality ratios by three- official and semi-official publications cite im- quarters between 1990 and 2015; provement of the status of women as a primary 6. Provide access for all who need reproductive social goal. health services by 2015, and Among the health indicators, life expectancy 7. Implement national strategies for sustainable at birth, infant and child mortality rates, and development by 2005 so as to reverse the loss maternal mortality ratios are all reported. of environmental resources by 2015. CAPMAS has recently completed a new set of Many of these goals and indicators to meas- maternal mortality estimates. In its health status ure the goals are incrporated in data sets used index, the Social Fund for Development uses by Egyptian agencies to monitor their programs. the infant mortality rate, the under-five mortal- A new household expenditure survey, from ity rate, and the maternal mortality ratio as its which poverty rates can be calculated, will be measures of human well-being. The Ministry of released shortly. How to measure poverty and Health has an extensive program of reproduc- the proper definition of the national povrety line tive health care through its primary health care is much debated. The 1996 Human Develop- units, which record fertility information and ment Report for Egypt reported on five poverty contraceptive prevalence in their service popula- lines: a food-based poverty line, a lower and an tions. They also collect information on water upper income poverty line, and a lower and an supply. upper expenditure poverty line. All of this is evidence that Egypt is actively Enrollment levels in primary and secondary engaged in monitoring progress along the same school are widely reported and Egypt is working dimensions of human well-being as the Interna- toward a goal of universal enrollment in basic tional Development Goals and has need of high (through grade 8) education. However, only quality data to do so. gross enrollments (including out-of-age stu- Annex D Egypt,"in Research for Development in the Middle East and North Africa. IDRC, Notes, References, and Resources www.idrc.ca/books/focus/930/15koraye.html. 3. Egypt Social and Structural Review, draft, The World Bank, 2001. Notes 1. Valsan, E.H. The Egyptian Civil Service and the References Continuing Challenge of Reform. In Research in Egypt: Human Development Report, Institute of Public Administration. Volume 5: 223-226, 1999. National Planning, 1998, 2000. 2. Korayerm, Karima. "The Research Environment in El Saiedi, H.E. Dr. Ali, Restructuring of the Power Annex II 199 Sector and Enhancement of Private Opportunities, The National Council for Women. Pamphlet, n.d. See Presentation to the American Chamber of Com- also http://ncw.gov.eg merce in Egypt, October 2000. Towards a More Result Oriented Budget Process, in Healthy Egyptians 2010, Ministry of Health and Egypt: Public Expenditure Review of the Social Population Sectors, World Bank, Social and Economic Devel- Korayerm, Karima. "The Research Environment in opment Group, Middle East and North Africa Egypt," in Research for Development in the Region, January 1999. Middle East and North Africa. IDRC, Valsan, E.H. The Egyptian Civil Service and the www.idrc.ca/books/focus/930/15koraye.html. Continuing Challenge of Reform. In Research in Public Debt Management Program, Arab Republic Public Administration. Volume 5: 223-226, 1999 of Egypt, Ministry of Finance, Presentation to the www.IDSC.gov.eg (web site of the Information and Euromoney Conference, Emerging Arab Decision Support Center of the Egyptian Cabinet) Economies: Breaking New Ground in the Global http://www.oecd.org/puma (web site of OECD's Pro- Markets, September 2000. gramme on Public Management and Governance) Annex III Millennium Development Goals (MDGS) List of Goals and Targets Goal 1: Eradicate extreme poverty and hunger Indicators Target 1: Halve, between 1990 and 2015, the 1. Proportion of population below $1 per day proportion of people whose income 2. Poverty gap ratio [incidence x depth of is less than one dollar a day poverty] 3. Share of poorest quintile in national con- sumption Target 2: Halve, between 1990 and 2015, the 4. Prevalence of underweight children (under- proportion of people who suffer five years of age) from hunger 5. Proportion of population below minimum level of dietary energy consumption Goal 2: Achieve universal primary education Indicators Target 3: Ensure that, by 2015, children every- 6. Net enrolment ratio in primary education where, boys and girls alike, will be 7. Proportion of pupils starting grade 1 who able to complete a full course of pri- reach grade 5 mary schooling 8. Literacy rate of 15-24 year olds Goal 3: Promote gender equality and Indicators empower women Target 4: Eliminate gender disparity in pri- 9. Ratio of girls to boys in primary, secondary mary and secondary education and tertiary education preferably by 2005 and to all levels 10. Ratio of literate females to males of 15­24 of education no later than 2015 year olds 11. Share of women in wage employment in the non-agricultural sector 12. Proportion of seats held by women in national parliament Goal 4: Reduce child mortality Indicators Target 5: Reduce by two-thirds, between 1990 13. Under-five mortality rate and 2015, the under-five mortality 14. Infant mortality rate rate 15. Proportion of 1 year old children immu- nised against measles 200 Annex III 201 Goal 5: Improve maternal health Indicators Target 6: Reduce by three-quarters, between 16. Maternal mortality ratio 1990 and 2015, the maternal mor- 17. Proportion of births attended by skilled tality ratio health personnel Goal 6: Combat HIV/AIDS, malaria and Indicators other diseases Target 7: Have halted by 2015, and begun to 18. HIV prevalence among 15­24 year old reverse, the spread of HIV/AIDS pregnant women 19. Contraceptive prevalence rate 20. Number of children orphaned by HIV/AIDS Target 8: Have halted by 2015, and begun to 21. Prevalence and death rates associated with reverse, the incidence of malaria and malaria other major diseases 22. Proportion of population in malaria risk areas using effective malaria prevention and treatment measures 23. Prevalence and death rates associated with tuberculosis 24. Proportion of TB cases detected and cured under DOTS (Directly Observed Treatment Short Course) Goal 7: Ensure environmental sustainability Indicators Target 9: Integrate the principles of sustain- 25. Proportion of land area covered by forest able development into country poli- 26. Land area protected to maintain biological cies and programmes and reverse the diversity loss of environmental resources 27. GDP per unit of energy use (as proxy for energy efficiency) 28. Carbon dioxide emissions (per capita) [Plus two figures of global atmospheric pollution: ozone depletion and the accumulation of global warming gases] Target 10: Halve, by 2015, the proportion of 29. Proportion of population with sustainable people without sustainable access to access to an improved water source safe drinking water Target 11: By 2020, to have achieved a signifi- 30. Proportion of people with access to cant improvement in the lives of at improved sanitation least 100 million slum dwellers 31. Proportion of people with access to secure tenure [Urban/rural disaggregation of several of the above indicators may be relevant for monitoring improve- ment in the lives of slum dwellers] 202 Annex III Goal 8: Develop a Global Partnership for Indicators Development Target 12: Develop further an open, rule-based, Some of the indicators listed below will be monitored sepa- predictable, non-discriminatory trad- rately for the Least Developed Countries (LDCs), Africa, ing and financial system landlocked countries and small island developing states. Includes a commitment to good governance, Official Development Assistance development, and poverty reduction--both 32. Net ODA as percentage of DAC donors' nationally and internationally GNI (targets of 0.7% in total and 0.15% for LDCs) Target 13: Address the Special Needs of the 33. Proportion of ODA to basic social services Least Developed Countries (basic education, primary health care, nutri- Includes: tariff and quota free access for LDC tion, safe water and sanitation) exports; enhanced programme of debt relief 34. Proportion of ODA that is untied for HIPC and cancellation of official bilateral 35. Proportion of ODA for environment in debt; and more generous ODA for countries small island developing states committed to poverty reduction 36. Proportion of ODA for transport sector in land-locked countries Target 14: Address the Special Needs of land- Market Access locked countries and small island de- 37. Proportion of exports (by value and exclud- veloping states ing arms) admitted free of duties and quotas (through Barbados Programme and 22nd Gen- eral Assembly provisions) Target 15: Deal comprehensively with the debt 38. Average tariffs and quotas on agricultural problems of developing countries products and textiles and clothing through national and international 39. Domestic and export agricultural subsidies measures in order to make debt sus- in OECD countries tainable in the long term 40. Proportion of ODA provided to help build trade capacity Debt Sustainability 41. Proportion of official bilateral HIPC debt cancelled 42. Debt service as a percentage of exports of goods and services 43. Proportion of ODA provided as debt relief 44. Number of countries reaching HIPC deci- sion and completion points Annex III 203 Goal 8: (continued) Indicators Target 16: In cooperation with developing 45. Unemployment rate of 15­24 year-olds countries, develop and implement strategies for decent and productive work for youth Target 17: In cooperation with pharmaceutical 46. Proportion of population with access to af- companies, provide access to afford- fordable essential drugs on a sustainable able, essential drugs in developing basis countries Target 18: In cooperation with the private sec- 47. Telephone lines per 1,000 people tor, make available the benefits of 48. Personal computers per 1,000 people new technologies, especially informa- tion and communications Annex IV National Evaluation Policy for Sri Lanka Sri Lanka Evaluation Association (SLEva) jointly with the Ministry of Policy Development and Implementation December 2003 adoption of a national policy on evaluation would National Evaluation Policy of the provide guidance and direction on the use of eval- Government of Sri Lanka uation and its role in national development. The current situation and the need for a Preamble National Evaluation Policy The Government of Sri Lanka fully recognises the Globally, public sector performance has been an growing international consensus that evaluation issue among citizens. Taxpayers have challenged is an essential aspect of good governance to im- governments to demonstrate value for money in prove development effectiveness, transparency, the provision of public service. The relevance of accountability and informed decision making. institutions and their mandates have been ques- The term `evaluation' in this document is referred tioned in a world of rapid change. Similarly with in the development context and the definition of regard to projects and programmes, the increas- the Development Assistance Committee ( DAC)/ ing share of problem projects and unsatisfactory OECD of "the systematic and objective assess- performance of completed projects emphasise the ment of an on-going or completed project, pro- need for systematic evaluation. Available evidence gramme or policy, its design, implementation and highlights that significant proportions of devel- results with the aim to determine the relevance opment programmes have failed to fully achieve and fulfilment of objectives, development effi- their envisaged development objectives. For ex- ciency, effectiveness, impact and sustainability'' is ample in Sri Lanka, it is reported that only 44% used. An evaluation should provide information of the Asian Development Bank funded post- that is credible and useful, enabling the incorpo- evaluated projects, have been successful in terms ration of lessons learned into the decision making of their contribution to the social and economic process. Systematic evaluation of projects, pro- development2. It has been widely accepted that grammes, institutions and policies is vital to im- timely evaluations and the use of reliable evalua- prove performance accountability, lesson learning tive knowledge help governments to improve pol- and policy refinement in the public sector1. Eval- icy and project designs, increase returns from in- uation is also a tool for public sector reforms. The vestments and speed up the implementation of ultimate success of evaluation depends on how on-going projects. well planners and decision makers use evaluation The government is conscious and mindful of findings and lessons learned to improve policy the fact that, at present, high proportion of mon- formulation and planning. Therefore it is neces- itoring and evaluation resources are devoted to sary to establish strong links between on the one monitoring the inputs and physical and financial hand evaluation, and on the other, policy formu- implementation of large projects and little atten- lation, reforms, planning and budgeting. The tion is devoted to assessing the results, sustain- 204 Annex IV 205 ability, delivery of services, the quality, distribu- formance monitoring and evaluation to policy tion of benefits of projects among various socio- through the reinvigorated National Operations economic groups or geographical regions. Moni- Room (NOR)3 of the Ministry of Policy Devel- toring and Evaluation System in the past tend to opment and Implementation (MPDI). Accessibly be more implementation biased, data rich, infor- of the development information to the policy mation poor and disbanded with termination of makers and general public is essential to ensure a projects. Evaluations in many cases are donor "functional National Operations Room" driven. Misperceptions of evaluations as policing or fault-finding exercises and lack of local de- National Evaluation Policy mand are other problems that inhibit the practice Objectives of the National Evaluation Policy of evaluations. In addition the mechanism to in- corporate evaluation findings into new project de- The National Evaluation Policy is intended to signs needs to the strengthened. These issues need achieve the following objectives. to be addressed immediately. a. Promote the correct understanding of evalua- The need to achieve results from public devel- tion and create an evaluation culture among opment interventions has become extremely im- the public sector managers to use evaluations portant with resource constraints and persistent to `manage for results'. development disparities resulting from ineffec- c. Promote the practice of evaluation through tiveness and inherent weaknesses of programmes. catalysing the generation of necessary human This pressure to demonstrate results has led to the and institutional capacities, tools and method- introduction of Results-Based Monitoring and ologies. Evaluation (RBME) system in the government d. Enable Learning of lessons from past experi- machinery. The need for planned and systematic ences to identify the policies, programmes, evaluation at all levels of government is therefore projects and delivery systems most likely to timely. This becomes even more crucial in Sri succeed and factors most likely to contribute to Lanka, at the present time when there is enor- that success. mous potential to move into a period of rapid de- e. To contribute to improve the design of devel- velopment--including the reconstruction of the opment policies and programmes through ef- war affected and adjacent areas--following the fective integration of evaluation findings into onset of the peace process, as well as the major the policy formulation, reforms, planning and economic reform programme under the "Regain- budgeting process. ing Sri Lanka" initiative. f. To enhance or promote accountability, trans- The formal adoption of a national evaluation parency and good governance. policy and its implementation would set up an enabling environmenttocontinuouslytrackprogress,reviewper- Fundamental Principles of National formanceandfinetunepolicies in order to realize the Evaluation Policy vision and aspiration of private sector led eco- The national evaluation policy is based on the fol- nomic development. Furthermore creation of a lowing fundamental principles. suitable policy environment for evaluations com- plements the tools package necessary for system- · Evaluations are practical assessments serving atic development monitoring and linking per- practical ends, neither scientific research stud- 206 Annex IV ies undertaken for advancement of knowledge potential learning content and development nor acts of policing. relevance. (See selection criteria below) · Evaluation should be seen primarily as an in- · Use of performance indicators and logical strument of accountability and lessons learn- framework based approaches should be made ing. Independence is of utmost importance for mandatory for all policy, programme or proj- objectivity and credibility of evaluation. How- ect preparation initiatives, thereby making it ever, participation needs to be built-in for les- possible to subsequently evaluate them mean- sons learning purposes. Depending on the ingfully. needs and circumstances, management should · The civil society organizations (CSOs), private justify the selection of external, internal or sector and academics should be encouraged to joint evaluations. All public sector institutions undertake evaluations preferably in partner- should be encouraged to use evaluations as in- ship with relevant public institutions. struments to account for the results as well as · The national and sub-national level execution lesson learning. authorities are responsible to ensure that disci- · All types of evaluations--Ex-post, impact and pline of evaluations is sufficiently deployed mid-term evaluations--that serve different pur- within their major cost centres. poses and, are conducted at different phases of It is emphasised that all public sector institu- the project cycle, need to be encouraged. Pref- tions embed evaluation into their development erence should be given to on-going and mid management practices. term evaluations. In order to have a wider per- spective of development, the government ac- Operationalization cords special attention to the evaluation of sec- tors, institutions, policies and thematic areas. Selection Criteria for Evaluation · The evaluation findings and lessons should be It may not be advisable to evaluate all develop- linked to the policy formulation, reforms, ment programmes for both practical and financial planning and budgeting process. The govern- reasons. The authorities responsible for the exe- ment should learn from evaluation findings cution of a given programme or project should and communicate and share evaluation infor- initiate action for undertaking evaluation of mation with other stakeholders. Findings of major projects, programmes and policies. In this evaluations of development policies and pro- regard all Sectoral Ministries should develop their grammes should be readily available to public own evaluation plan. Further, the Economic Pol- and media. Specific website(s) may be main- icy Committee and/or NOR and/or National tained for this purpose. Level Monitoring Committees should identify · Evaluations should be made mandatory under areas for evaluation on a rolling biennial plan. certain circumstances and adequate provision When selecting projects and programmes for eval- should be made up-front. The policy, pro- uation on their own, the Sectoral Ministries may gramme or project proponents should ensure form a team comprising of the following; evaluability. Concerns for evaluation should be built-in at the time of planning and design of 1. Representative of the Monitoring and Progress programmes and policies. Subjects to be eval- Review Division (MPDI) of the MPDI or NOR. uated should be selected on the basis of their 2. When foreign funded projects are to be se- Annex IV 207 lected, representatives of the Department of cies involved in national development work. The External Resources and funding agency. MPDI shall provide necessary assistance and 3. Representatives of the Department of National guidelines, training and refresher courses regu- Planning and Department of National Budget. larly to implement the national evaluation policy 4. Representatives of the academia and/or civil more effectively. The Central Performance Evalu- society organizations such as Sri Lanka Evalu- ation Unit (CPEU) of the MPDI will serve as a ation Association (SLEvA). focal point for implementing the National Evalu- ation Policy. The secretaries of the line ministries This committee should screen and select suit- should be responsible for the development of an able projects for evaluations. This will enable the evaluation plan in their respective sectors or areas. MPRD/ MPDI to maintain track of the current Each Sectoral Ministry when initiating inde- evaluation studies. The following criteria should pendent evaluations within their own areas of re- be used by the selection committee when selecting sponsibility, should in consultation with MPDI, projects or programmes for evaluation. develop evaluation plans and terms-of-reference. 1. Policy relevance e.g. poverty reduction, Ministries may obtain the services of public sec- 2. National importance and the scale of funding tor, private sector, universities, CSOs and indi- 3. The innovative value and replicability of proj- vidual professional evaluators to undertake such ect or programme. In this context some `small' studies. The respective sectoral Ministry in con- projects could also be evaluated. sultation with the CPEU of the MPDI and other 4. Public interest and problem nature relevant stakeholders should develop the terms of reference (TOR) for such evaluations. The Cen- Evaluations should not only cover problem tral Evaluation Unit on the other hand is respon- areas but also draw lessons on success stories. sible for more comprehensive and strategically im- With the change in the role of government as a fa- portant evaluation of a thematic nature. cilitator and the need to finance and execute pub- MPDI in close collaboration with professional lic investments by private sector, it is necessary to evaluation associations develop and promote encourage private sector to undertake independ- evaluation culture, standards, guidelines, method- ent evaluations of their own activities - specially ologies, best practices, monitor and develop eval- investments of a public nature. These evaluations uation capacity, sensitise policymakers, and facil- should be carried out through the representation itate the dissemination of evaluation findings. bodies such as Chambers. Such evaluations are The evaluations initiated by the Sectoral Min- necessary to demonstrate private sector's contri- istries would tend to be more of leaning exercises bution to the national development and to in- while those conducted by the Central Evaluation crease public accountability and transparency. It Unit of the MPDI would tend to be more ac- is also necessary for the evaluation to focus its at- countability and policy influence oriented. Some tention to policies connected with private sector form of compromise and balance is needed be- rather than purely projects. tween accountability and lessons learning. Implementation of National Evaluation Policy Dissemination of Evaluation Findings Implementation of the National Evaluation Pol- Each sector that undertakes an evaluation should icy is the responsibility of all Ministries and Agen- also develop a dissemination strategy for sharing 208 Annex IV lessons internally and as well as externally. All Sec- programme initiatives. Evaluation methodology toral Ministries should forward reports of evalu- should look into the financial, economic, social ations to the CPEU of the MPDI (especially elec- (including conflict sensitivity), environmental, gen- tronically). This will enable evaluation findings to der, institutional and sustainability aspects. The be synthesized and linked to the Evaluation Infor- use of financial and economic cost benefit analy- mation System (EIS) of the NOR of the MPDI and sis to assess the value for money, need to be en- Economic Policy Committee (EPC) to ensure in- couraged. Moreover, the evaluation methodology tegration of evaluation findings into the policy, should integrate social and environment concerns. planning, budgeting and reform processes. Eval- Beneficiary assessment should form an integral uation information should be made accessible to part of evaluating social programmes. Due con- the parliament, national audits and general pub- sideration should be given to the political and pol- lic. The Sectoral Ministries should after the com- icy environment. Concerns for evaluation should pletion of the evaluation dissemination workshop, be integrated at the time of planning and formu- prepare a plan of action, which should identify the lation of the project. Use of programme theory, specific follow-up action with well defined time logic model or Logical Framework Analysis (LFA) scales and responsibilities. Copies of such plan of with well-defined performance indicators at the action and the progress should be submitted to the time of project preparation is mandatory for proj- MPDI. MPDI and the Line Ministries are respon- ects which are over US $ 10 million. Projects less sible to ensure the implementation of the plan of that US $ 10 million should also be encouraged to action and ensure that evaluation funding, lessons use such approaches whenever possible with base- and follow-up actions are integrated into the de- line and benchmark indicators. The CPEU with velopment planning and management framework. SLEvA and other professional CSOs are encour- The project proponents and the national plan- aged to proactively participate in the preparation ning authorities should ensure the incorporation of major projects by reviewing and confirming the of evaluation findings in the formulation of new performance indicators, both baseline and targets. projects and programmes. For this purpose the As evaluations are practical investigations and not evaluation findings and lessons should be pre- scientific research studies, simple, cost effective sented in a brief, reader friendly summary. The and less time consuming participatory rapid ap- project submission formats and related proce- praisal methodologies may be used preferentially. dures should be suitably modified to internalise It is also necessary to develop local evaluation evaluation findings into the planning, budgeting, methodologies, guidelines, standards, ethics and public expenditure review, and policy formulation practices in par with accepted international stand- process. In this regard a close collaboration should ards. Evaluation methodology, knowledge and ex- be established among, evaluation, planning, budg- pertise should be developed to take into account eting, audit, finance, public expenditure and pol- the specific needs of sectors. The MPDI in collab- icy review functions of the government. oration with the Sri Lanka Evaluation Association and other CSOs should undertake this task. Guidelines, Methodologies, Standards and Ethics Capacity Building and Partnerships Both on-going as well as ex-post evaluations should examine the relevance, efficiency, effec- The availability of adequately skilled competent tiveness, impact and sustainability of policy or human resources in evaluation is essential. Gov- Annex IV 209 ernment recognises the need to build a professional in-country participation. Such unilateral ap- cadre of evaluators and accords high priority for proach, though helps to ensure objectivity of eval- capacity building efforts. The Universities and pub- uation, does not assist in the development of in- lic sector training institutions should be encour- country capacities nor does it help to linking the aged to run evaluation modules as part of their nor- evaluation to overall planning process. Govern- mal programmes. The government would also ment should encourage donors to strengthen in- encourage joint evaluations and regional network- country evaluation capacity. Moreover, all evalu- ing to share knowledge on evaluation techniques ation missions on foreign funded projects and and methodologies. Joint evaluations while ensur- independent evaluations should have links with ing independence would also help to establish the CPEU to ensure central coordination on eval- local ownership and in-house capacity building in uation. A documentation centre should be in evaluation. By end 2005, it is envisaged that all place at CPEU to access all the evaluation reports. major evaluations should have significant national Consultants and Contracting ownership. Local participation should be ensured in planning, designing, implementation and dissemi- The Sectoral Ministries shall select qualified, com- nation of evaluation to enhance local ownership. petent and experienced professional firms or in- Sectoral ministries should strengthen the ca- dividuals whenever possible locally. The govern- pacity for performance evaluation, ex-post evalu- ment is committed to promote domestic capacity ation and impact evaluations in their area of re- in evaluation. Joint ventures between domestic sponsibility. The MPDI must provide central evaluation professionals and foreign consultants direction for evaluation and should (a) upgrade should also be encouraged to transfer knowledge the CPEU as a centre of excellence to provide and skills on evaluation methodologies, tech- leadership, guidance and support to the practice niques and practices. of evaluation; (b) use evaluation findings where Financing Evaluation appropriate in decision making; (c) set standards, ethics and best practices and (d) monitor the eval- It is necessary to have sufficient financial re- uation capacity in the public sector. sources for conducting evaluations of an accept- The CPEU of MPDI jointly with professional able quality. Ministries and Provincial Councils civil society evaluation organizations will assist should make necessary provision in the annual sectoral Ministries to build evaluation capacity, budget estimates for the conduct of evaluations. develop standards, methodologies and upgrade In addition to the financial support under the con- capacity of their staff. As part of the efforts to solidated funds of the government, it is also nec- build local evaluation consultancy industry, the essary to have built-in-funds under foreign aided Sectoral Ministries, may outsource evaluation projects for the conduct of evaluations. It is nec- work to private sector and civil society organiza- essary for the government to provide regular tions (CSOs). Government will encourage such funding for post-evaluations, which cannot be collaboration and partnership with NGOs and generally built into the foreign funded projects. CSOs to introduce participatory evaluations in Similarly financing arrangements should be made the public sector. for institutional, policy and thematic evaluations. Many donor funded post evaluations have been There should be a separate special vote under the conducted by donors themselves without much MPDI and other line ministries for such purposes. 210 Annex IV Oversight Notes The MPDI will monitor the implementation of 1. Utility of evaluations applies also to other sectors. this policy to ensure its success in meeting the in- This document however is focused on national tended objectives. Secretary, MPDI in close con- policy for the public sector. It may be applied in sultation with the professional CSOs such as other sectors as desired. SLEvA, Chamber of Commerce, Organization of 2. Country synthesis report on evaluation of the Asian Development Bank, 1999. The performance Professional Association, will monitor the imple- of projects funded by other donor agencies is not mentation of the policy every year. A consultative known to be any better. and oversight modality, which would, inter alia, 3. National Operations Room (NOR) which will be reflect the creation of the evaluation culture in the focal point to collect, analyse and present eco- the public sector would be developed for this pur- nomic and development information policy mak- pose by the MPDI in consultation with the stake- ers and monitor and evaluate national and sub- holders. national development work and investments. Annex V Andhra Pradesh (India) Performance Accountability Act 2003 (Draft Act) (APPAC Act of 2003) Preamble Chapter 8 Monitoring & Evaluation Chapter 9 Incentive & Disincentives An Act to enhance accountability, manage Infor- Chapter 10 Apex Committees mation Systems, evaluate performance of individ- Chapter 11 Annual Performance Reports uals, Departments and Institutions in the State of Chapter 12 Human Resource Developments Andhra Pradesh and for all matters connected Chapter 13 Smart Governance there with or incidental thereto. Chapter 14 Miscellaneous The State of Andhra Pradesh is poised for Appendix A Good Governance with efficient management of Appendix B all resources and moving towards being a Swar- nandhrapradesh. Where the scientific and systematic develop- Chapter 1 ment of the state can be best possible through ef- ficient management of Information Systems em- 1. An Act anating from Gross root level. Such development To provide for the establishment of strategic plan- seeks for a need based evaluation of the perfor- ning and performance measurement in the State mance of Individuals, Departments and Institutions. Government and for other purposes. And where the individuals, Departments and Be it enacted by the State Legislative Assembly Institutions shall be accountable for their per- of the Government of Andhra Pradesh (GOAP). formance with incentives and dis-incentives. Such a system of accountability and evaluation 2. Short Title shall establish SMART Governance (Simple, Moral, This Act may be cited as "The Andhra Pradesh Accountable, Responsive and Transparent). Performance Accountability Act 2003". It shall extend to the whole of the State of Contents Andhra Pradesh including a) All Departments under the State Govern- Chapter 1 Short Title ment; Chapter 2 Findings and Purpose b) All Semi Government bodies, Local bodies, Chapter 3 History of Administrative Reforms Co-operative Institutions etc., under the in AP control of the Government; Chapter 4 Performance Accountability System c) All Public Sector Institutions under the con- (PAS) trol of the Government; and Chapter 5 Strategic Planning d) All Organizations or Institutions or individ- Chapter 6 Information Flow uals receiving any form of grant or assis- Chapter 7 Performance Measurement and tance or aid, whether monetary or otherwise Documentation from the Government or public funds. 211 212 Annex V (1) It shall come into force on such date, as the them with information about service qual- Government may, by notification in the ity and results; Andhra Pradesh Gazette. (e) Improve decision making at various levels by providing more objective information Chapter 2 on achieving goals, improving effectiveness and efficiency of Government programs Findings and Purposes and spending; and 1. Findings (f) Improve the internal management of the The Government finds that: State Government. (a) Lack of efficiency in State-run programs undermines the confidence of people and Chapter 3 reduces Government's ability to address ad- History of Administrative Reforms in AP equately issues of public interest; (b) Functionaries in the Government are dis- The State of Andhra Pradesh has been a pioneer advantaged in their efforts to improve the in initiating Administrative Reforms for improv- program efficiency and effectiveness, due to ing the performance of State run Programs inadequate information flow; and 1. K. B. Lal Ananthraman & Sriramulu Com- (c) Policy making and financial decisions are mittee on Administrative Reforms (1976) seriously handicapped by insufficient artic- 2. M. K. Rustomji & Associates on Adminis- ulation to programme goals and objectives, trative Reforms (February 1986) performance and results. 3. Action Plan for Administrative Reforms (June 1986) 2. Purposes 4. Committee on Administrative Reorganiza- Purposes of this Act: are to: tion--S.R. Ramamurthy, G.R. Nair and (a) Improve confidence of the people, in the ca- K.V. Natarajan (April 1990) pability of the State Government by sys- 5. Staff Review Committee--B.C. Gangopad- tematically holding the Government De- hyaya & J.M. Girglani (April 1994) partments, Institutions and individuals 6. Cabinet Sub-Committee on Administrative accountable for achieving program results; Reforms--headed by Sri Devender Goud (b) Initiate a series of performance reforms by (January 1997); setting up program goals, measuring per- 7. Three officers Committee on formance against those goals, and report- a) Reorganisation of Secretariat Depart- ing publicly on their progress; ments (M.V.P.C. Sastry); (c) Improve Government effectiveness and b) Reorganisation of Commissionarates and public accountability by focusing on re- Heads of Departments (N.S. Hariharan); sults, service quality, and customer satis- c) Delegation of powers to District collec- faction; tors etc (B. Danam) (d) Motivate Government functionaries to im- 8. Special five-member Committee in each de- prove service by orienting them for plan- partment headed by the Secretary concerned ning to achieve objectives by providing (December 1997); Annex V 213 9. Task Force on Good Governance--headed 4) At the beginning of each Financial Year, the by Sri Madhav Godbole (January 2000) heads of the departments shall prepare a 10. Cabinet Sub-Committee on Administrative plan for the year in consultation with the Reforms headed by Sri Vidyadher Rao Secretary and the concerned Minister. This (2000) plan shall be submitted to the Minister to be 11. Strategy paper on Governance and Public in turn presented in the assembly. Management (January 2002) 5) The functions and activities of the strategic plan shall be inherently Government func- Chapter 4 tions and they shall be performed only by Performance Accountability System (PAS) the Government functionaries. Performance Accountability System shall be es- 2. Contents of a Plan tablished in each Department or Institution com- The strategic plan shall contain prising of a comprehensive framework of Perfor- 1) A comprehensive mission statement cover- mance Management Activities including: ing the major functions and operations of 1. Strategic Planning the department to enhance policy making 2. Information Flow capability in Government and to improve 3. Performance Measurement the performance of the key parts of the pub- 4. Performance Monitoring and Evaluation lic service which contribute significantly to 5. Performance Budgeting the social and economic development of the state; Chapter 5 2) A Vision statement indicating the direction Strategic Planning in which the Department intends to move 1. Strategic Planning and what are the major achievements that it Every Government Department or Institution aims at; shall draw up a strategic plan which shall be in 3) General goals and objectives, including out- congruence with the Vision of the State Govern- come-related goals and objectives for major ment. functions of the department 1) It shall focus on (a) the baseline, (b) Identify 4) A description of how the goals and objec- benchmarks (c) spell out objectives and tives are to be achieved (Action Plan), in- strategies; cluding a description of the operational 2) The strategic plan shall cover a period of processes, skills and technology and the one year from the fiscal year in which it will human, capital, information and other re- be submitted, and shall be updated and re- sources required to meet those goals and ob- vised; jectives 3) While developing a strategic plan, the Gov- 5) A description of how the performance goals ernment shall be consulted and the views included in the plan shall be related to the and suggestions of those potentially affected departmental goals by or interested in such a plan may also be 6) A description of various levels of accounta- taken. bility, i.e.; the measurable goals 214 Annex V 7) An identification of those key factors exter- analysis, documentation, retrieval, preserva- nal to the agency and beyond the control tion and communication of information, as- that could significantly affect the achieve- sisted by his subordinate officers and staff. ment of the general goals and objectives and 8) Comprehensive description of the evalua- 3. Explanation tion tools used in assessing or revising the (a) In case, where any Department or Institution objectives and formulation of compatible has no District level office within the total goals jurisdiction of the District or forms a part of the District or spreads over two or more Dis- Chapter 6 tricts, it shall be identified as the unit office. Example: A circle office in an Engineering Information Flow Department shall be the unit office for the 1. Classification of Information purpose of this clause. (1) All information called for, from any source (b) In case, where any Department or Institu- shall be either coded, verbal, textual, nu- tion has no Mandal level office within the merical, audio-visual, alpha-numerical, total jurisdiction of the Mandal or forms a graded or percentages or any other kind as part of the Mandal or spreads over two or prescribed and shall be in the prescribed more Mandals, it shall be identified as sub- formats, specific to each level, in each De- unit office. partment or Institution. Example: A division office in an Engineer- (2) The information shall be classified as Or- ing Department shall be the sub-unit office dinary, Urgent, and Top Priority and shall for the purpose this clause. be designated as X, XX, XXX in the for- (4) All the individuals working in the area of mats and confidential information shall be operation at each information centre shall designated as `Confidential'. personally be accountable for assisting the nodal Officer at that particular level for 2. Information centers submission of systematic and periodic in- (1) There shall be three main levels of infor- formation to the next higher level. mation centers i.e. (a) State level, (b) Dis- trict / Unit level and (c) Mandal / Sub-unit 4. Mode of Communication level for each Department or Institution. (1) The mode of communication of informa- (2) Each Department or Institution shall iden- tion from one level to another level shall be tify and designate the three levels as speci- predetermined with approved process ei- fied in subsection (1) for communicating ther through verbal, personal, telephonic, the information from lower to higher level telegraphic, wireless, postal, electronic or and shall be notified in the Gazette. any other prescribed media of data com- (1) There shall be one nodal officer (by designa- munication systems. tion) at each level, who shall be the head of (2) It may be with one or more modes as given the office of that unit and shall be personally in sub-clause (1) and shall be specified in accountable for collection, compilation, the prescribed formats. Annex V 215 5. Formats 6. Periodicity of Information flow (1) The collection, compilation, analysis, doc- (1) The periodicity of flow of information from umentation, preservation and communica- one level to another level may be online, tion of any kind of information shall be in hourly, daily, weekly, fortnightly, monthly, the prescribed formats, specific to each De- quarterly, and half yearly or yearly as pre- partment or Institution and shall be ap- scribed in the individual formats. proved by the Apex Committee as pre- (2) All other correspondence in which any in- scribed in the Section ...... formation is called for from the subordi- (2) Each format shall be coded with nine digit nate offices, other than those called through code covering Department code (three digits) the approved formats as specified in sub- Information classification code (three digits) section (1) shall contain invariably the date and individual format number (three digits). and time of receipt of information and the Explanation: mode of communication through which the (a) For the purpose of Management of In- information shall be sent to the such higher formation Systems, each Department or office and vice-versa. Institution shall be given a specific code 7. Power to obtain information number in three digits (example: 036) (1) The Apex Committee as specified in Section (b) Information classification code shall be ...... in each Department may, with a view specific to each Department or Institu- to achieve the objectives of the Act shall call tion and shall be in three digits (ex- for any information from individuals and ample 027) all unit and sub-unit offices in their juris- (c) The format number shall be the serial diction, in the prescribed formats. number of the format under each Infor- (2) The Apex Committees shall also have the mation classification code in three dig- power to call for any other information re- its (example: 054) lated from other Departments or Institutions (3) The formats shall be designed to extract the and it may be at the Apex committee level. right information suitable for analysis and (3) The Government shall have the power to amenable for computerization. call for any information from any Depart- (4) It shall specify periodicity, the designation ment or Institution and also from any of of officer (to authenticate the information), the citizens in the state in connection with mode of communication etc., along with the services rendered through different De- necessary instructions thereon for collec- partments and agencies of the Government. tion, compilation, analysis, documentation and communication of information. Chapter 7 (5) The formats specified under this section Performance Measurement and Documentation shall be periodically reviewed and updated and such updated formats shall have the 1. Performance measurement codes with suffix alphabets in succession Performance measurement shall connect the for each updation. strategic plans to results and shall be a continu- (Example: 036 027 054 A ) ous process of 216 Annex V 1) Performance Achievement through a set of (2) There shall be a documentation officer des- Indicators i.e. achievement of monthly and ignated for the purpose under the control cumulative physical and financial targets-- of each nodal officer who shall be person- indicators, functionaries, institutions & ter- ally accountable for documentation, preser- ritorial jurisdiction; vation and retrieval of the information and 2) Progress of important projects (physical and shall take all measures for the safety, secu- financial) rity of all records and for retrieval of infor- 3) Achievement of process targets in relation to mation at any given time, whenever called benchmarks and best practices at all levels for. of Government (3) The procedure for documentation shall be 4) Collating Information i.e. the information as prescribed. received at each information centre shall be 3. Research and Analysis Wing (RAW) abstracted and such abstracted information (1) There shall be one Research and Analysis shall flow from lower information centre to Wing (RAW) at the state level information the next higher centre in the prescribed for- center in each Department or Institution to mats in the given time schedule. analyze historic and current data received 5) Introduction of an Online Performance from various information centers and other Tracking System sources. 6) Measurement of Results (2) It shall be headed by one of the senior offi- 7) Identifying Success/Failure cers of the Department or Institution with complementary staff and shall be under the Explanation: control of the Secretary of the Department The Mandal / Sub-unit shall abstract the infor- concerned. mation at their level and send to the information (3) It shall periodically analyze the information Center at the District / Unit in one or more modes to draw out the trends for policy making by as specified in the formats; and so on. the Government in each Department on its The analysis of information shall be either objectives, as prescribed. manual or through computer or any other means (4) Subject specialists may be associated with as specified in the formats. the Departmental officers and staff in the The information officer at each information cen- Research and Analysis Wing. ter shall be personally accountable for the analysis (5) The procedures and the functions of the and shall authenticate the abstracts before sending Research and Analysis Wing shall be as them to the next higher information center. prescribed. 2. Documentation Chapter 8 (1) All information collected, analyzed shall be Monitoring and Evaluation documented through paper or electronic or any other prescribed media before the next 1. Evaluation information is received in the periodicity at (1) Evaluation of performance shall be on each the information center. individual working in the jurisdiction of Annex V 217 three level centers i.e. (a) state level centre State Level -- Secretary or Head of the (b) District/Unit level (c) Mandal/Sub-unit Department level a nd in each Department or Institution (2) The Chief Minister or the Chief Secretary in the State. of the State shall review performance of all (2) Performance indicators shall be evolved for Departments or Institutions at Government each level as specified in Section (1) per- level assisted by one or more Secretaries taining to their jurisdiction in the pre- designated for the purpose. scribed formats approved by the Apex (3) The Authority to review the performance of Committee in each Department or Institu- all other Organizations or Institutions or tion and may be periodically revised by the individuals receiving any form of grant or Government, as per necessity. assistance or aid, whether monitory or oth- 2. Evaluation parameters erwise from Government or Public funds (1) The parameters for evaluation of individu- shall be as prescribed. als, Departments or Institutions shall be, as approved by the concerned Apex Commit- 4. Review Meetings tees or the Government, as the case may be. (1) The Officer In charge at Mandal / Sub-unit (2) The parameters shall include indicating the shall review the performance in his juris- performance of individuals, Departments or diction in the prescribed formats, once in a Institutions for revenue recovery, economy in month (i.e.) 1st day in every month. expenditure, usefulness in expenditure, skills (2) The District Collector or the Unit Officer of in planning, time management, achievement concerned Department or Institution shall of goals, quickness of disposals, Adminis- review the performance in his jurisdiction trative skills, monitoring and inspection, or in the prescribed formats, once in a month such other parameters as prescribed. (i.e.,) on 5th of every month. (3) All parameters shall be scalable or quantifi- (3) The Secretary or the Head of the Depart- able with time, money, work wise etc., and ment as the case may be, shall review the shall be specified for individuals, Depart- performance of the Department or Depart- ments or Institutions in a scientific manner. ments or Institutions as the case may be in (4) There shall be four grades for evaluation of the prescribed formats, once in a month the performance for each individual De- (i.e.) on 10th of every month. partment or Institution (Example: A, B, C, (4) The Chief Minister or the Chief Secretary and D) as approved by the Government shall review the performance of all the De- 3. Evaluation Authority partments in the 2nd week of every quarter (1) The following officers shall review the per- (i.e.) during the month of January, April, formance of individuals working in their ju- July and October every year. risdiction with the prescribed indicators. (5) Special review meetings for any purpose Mandal / Sub. Unit level -- Officer In shall be conducted at any level, mentioned charge of Mandal/Sub Unit under this section, in addition to the regu- District / Unit level -- District Collector/ lar review meetings, as per the Government Unit Officer Orders from time to time. 218 Annex V (6) The procedure for review meetings and the 3. Appellate Authority accountability for the individuals shall be The appellate authority on the orders of the as prescribed. Apex Committee or the Government as the case may be under this Act shall be the High Court of Chapter 9 Andhra Pradesh. Incentives and Disincentives Chapter 10 1. Incentives Apex Committees 1) Incentives for individuals, Departments or Institutions for their high performing may 1. Formation of Apex Committee instituted as a special recognition and it (1) There shall be one Apex Committee for shall be as specified. each Department or Institution headed by 2) The Apex Committee shall be the Author- the Minister in charge of the Department or ity for awarding such incentives to individ- Institution as Chairman and the Secretary uals in their jurisdiction, as per the norms or Head of the Department concerned as of the Government. the Vice-Chairman. 3) The Government shall be the Authority for (2) The Apex committee shall be a nine mem- awarding such incentives to the Depart- ber committee including the Chairman and ment or Institutions. Vice-Chairman along with seven other members (by designation) drawn from dif- 2. Disincentives ferent activities like Administration, Ac- (1) In the case of non-performance, discipli- counts, Technical etc., in that Department nary action shall be initiated by competent or Institution. authorities by way of penalties as pre- (3) The Apex Committee may co-opt experts, scribed under CCA Rules for the following, up to a maximum of six members or invite under this Act. any guest members for suggestions. a) Non submission of information (4) The Apex committee shall be for a period b) Consistent delay in submissions of in- of three years and any casual vacancy shall formation be filled in by the Apex Committee by co- c) Submission of false information opting members d) Inaccurate analysis (5) The function of the Apex Committee shall e) Breach of official duty in connection be as prescribed and shall be the Authority with the Accountability fixed under this for implementation of the Act in their Ju- Act and Rules. risdiction of the Department or Institution. f) Failure of performance as prescribed g) Financial irregularities h) Misuse of Stores or Tools and Plant. 2. Meetings i) Failure to convene review meetings, fol- (1) The Apex Committee in each Department low-up actions, Inspections monitoring or Institution shall meet compulsorily on etc., 10th day of every quarter i.e. during Janu- j) Failure for documentation and preser- ary, April, July and October every year. vation of records. (2) The meeting shall be chaired by the Chair- Annex V 219 man for every meeting or by the Vice- every Department or Institution for its Chairman in the absence of the Chairman. smooth working to achieve its objectives. (3) The Chairman at his discretion may spe- (2) There shall also be a Function manual for cially convene the Apex Committee meet- each type of job with detailed procedures ing during other times, as per the need. and accountability at different levels in (4) One of the Apex Committee members shall each Department or Institution. act as convenor for organizing the meetings (4) All the individuals working in the Depart- who shall be nominated by the Chairman. ment or Institution shall be governed by the (5) The convenor shall be personally responsi- Department manual and Function manual. ble for convening the Quarterly meetings and special meetings on the dates specified 3. Job charts and shall document all minutes of the meet- (1) There shall be a job chart for every post, ing and the action taken reports. specific to its situation in each Department (6) The convenor shall initiate all actions nec- or Institution and the individuals working essary for implementation of the minutes of in that post shall be accountable for his per- the meeting and shall also bring all matters formance as per the job chart. in respect of the actions taken and to be (4) Separate job charts shall be prepared for taken on the minutes of the meeting to the the same post in different environments notice of the Apex Committee for further specific to its situation. action every time. Explanation: An individual though holds the same post, Chapter 11 may work in different environments which Annual Performance Reports requires some specific jobs to be performed Example: An engineer in the construction 1. Performance books unit, survey unit, water management unit, (1) There shall be a performance book for designs unit will have different jobs to per- every individual which shall be maintained form and so shall have different job charts by the officer in charge and handed over to specific to his situation. his successor at the time of change in in- (5)The preparation of job charts, scope and cumbency at all the three levels. fixation of responsibilities shall be as pre- (2) It shall contain self appraisal of the con- scribed. trolling officer as well as the appraisal of the subordinates on set goals and achieve- 4. Check lists ments for each month. (1) There shall be prescribed checklists for every (3) It shall be reviewed by the Departmental transaction requiring sanctions or approvals, promotion committees or officers at the either technical, monitory or administrative time of promotion to next higher cadre and in every Department or Institution. it shall be an open book, not confidential. (2) The check lists shall be a combined check- list for submission, processing and sanc- 2. Department manuals and Function manuals tion/ approval levels and shall be duly au- (1) There shall be a Department manual for thenticated by the individuals at various 220 Annex V levels and they shall be accountable at their for distribution among various Depart- level. ments, as a percentage on budget estimates, as prescribed from time to time. 5. Year book 4) The nodal training Institute shall develop a (1) Every information center shall, for the pur- strategic plan for all the training Institutes pose of efficient discharge of its functions based on the training needs assessment for under this Act, and guide that centre for officers and staff in various Departments subsequent year of operation shall docu- and Institutions. ment in typed script or in any other mode as prescribed regarding: 2. Trainings (a) Budgetary details; 1) The Trainings for officers and staff may be (b) All sanctions and approvals; imparted for Administrative skills, Techni- (c) Full details of receipts and cal skills, Management skills, and other expenditure; skills as prescribed, for improving the effi- (d) Incumbency of officers and staff; ciency and effectiveness for achieving the (e) Details of all activities in its program objectives of the state. jurisdiction; and 2) The Trainings may be of the following cat- (f) Other information as prescribed from egories and may be imparted during the time to time. tenure of the office. (2) The documentation officers concerned shall (a) Orientation Trainings; be personally responsible for preparation of (b) In-service Trainings; this year book by 1st May every year. (c) Special Trainings; and (3) The modalities for preparation of the year (d) Other Trainings as prescribed. book, distribution, preservation etc., shall 3) The training components, duration, num- be as prescribed. ber of trainings, selection of participants and other modalities required for organiz- Chapter 12 ing the training programs shall be, as pre- Human Resources Development scribed. 1. Training Institutes 3. Feed back analysis 1) There shall be a nodal training Institute at The effectiveness of the training shall be moni- the State level for Human Resources De- tored and accountability fixed on the trainees velopment, for officers and staff and Dis- based on the feed back analysis from trainees dur- trict training centers under its control. ing and after training. 2) The nodal training Institute shall monitor and coordinate the activities of all other Chapter 13 training Institutes in various Departments, as prescribed. SMART Governance 3) There shall be a central training Budget to Government of Andhra Pradesh is working to- be operated by the nodal training Institute wards a SMART Government Annex V 221 S: Simplifying Government: To enable Gov- bly of the state and shall be subject to such ernment to improve quality of service to the modifications by way of amendments or re- customer and increase his value for money peal as the Legislative Assembly may make through simplifying procedures. either in the same session or next session. M: Moral Government: To develop an effective HRM Plan by embedding new structures 2. Power to make Rules. and approaches to HRM (1) The State Government may, by notification A: Accountable Government: To improve the in the official Gazette, make rules to carry quality and timelines of delivery of services out the purpose of this Act. and to develop a flexible result-focused per- (2) Every Rule under the Act shall immediately formance culture across the public service after it is made, be laid before the Legisla- through systems which effectively monitor tive Assembly of the State, if it is in session and measure performance. and if it is not in session in the session im- R: Responsive Citizen Focused Government: mediately following for a total period of To ensure people have a strong voice in the fourteen days which may be comprised in governance of the state, through participa- one session or in two successive sessions, tory mechanisms into planning and moni- and if, before the expiration of the session toring of service delivery, enhancing decen- in which it is so laid or the session imme- tralization and ensuring inclusiveness of the diately following, the Legislative Assembly poor and disadvantaged agrees in making, any modification in the T: Transparency in Government: To improve rule or in the annulment of the rule, the rule planning, resource allocation, monitoring, shall, from the date on which the modifi- management and accounting systems and cation or annulment is notified, have effect access to information so that accountability only in such modified form or shall stand is clear, spending is transparent and public annulled as the case may be, so however, expenditure is more effectively controlled. that any such modification or annulment shall be without prejudice to the validity of Chapter 14 anything previously done under that rule. Miscellaneous 3. Protection of Actions done in good faith 1. Power to remove difficulties No penalty shall be levied against an individual, De- (1) If any difficult arises in giving effect to the partment or Institution to discharge any function provisions of this Act, the Government as under this Act, for any loss or damage caused or the occasion may require, by order pub- likely to be caused by any action which is in good lished in the Andhra Pradesh Gazette, do faith done or intended to be done in pursuance of any thing which appears to them necessary this Act or under the Rules made thereunder. for removing the difficulty. (2) All orders made under this section shall as Some Definitions soon as may be, after they are made, be In this Act, unless the context otherwise requires, placed on the table of the legislative Assem- 1. `Government' means the State Government 222 Annex V 2. `Department' means any State Government Institutions based on the approved reports Department under the Control of Govern- of Statutory Committees instituted for the ment of Andhra Pradesh. purpose. 3. `Institution' means an Institution estab- 7. `Information' means information either lished under law by the State or any other coded, verbal, textual, numerical, alpha- Institution which receives any form of grant numeric, audio-visual, graded, percentages, or assistance or aid, either monetary or oth- etc., generated or to be generated by indi- erwise from the Government. viduals, Departments and Institutions (both 4. `Individual' means an individual working in basic data and analyzed data) in perform- the Department or Public Institution and ance of duties. receiving salary or any form of remunera- 8. `Information Systems' means the approved tion or assistance from the Government or system of collection, compilation, analysis, public funds. documentation; retrieval and communica- 5. `Incentives' means all kinds of incentives ei- tion of the information from Gross root ther monetary, commendatory, promotions, level to Apex level, as prescribed. awards etc., given for the rated perform- 9. `Performance' means all kinds of scalable ance of individuals, Departments or Insti- performance in respect of achieving the ob- tutions based on the approved reports of jectives for the set goals, either monetary, the Statutory Committees instituted for the service, or other wise as prescribed. purpose. 10.`Notification' means notification published 6. `Disincentives' means all kinds of disincen- in the Andhra Pradesh Gazette and the word tives either monitory, condemnatory, de-pro- notified shall be construed accordingly. motions, penalties etc., given for the rated performance of individuals, Departments or Annex VI Glossary OECD Glossary of Key Terms in Evaluation and Results-Based Management (2002) Accountability: Obligation to demonstrate that sents an appropriate use of corporate re- work has been conducted in compliance with sources. agreed rules and standards or to report fairly Related term: ex-ante evaluation. and accurately on performance results vis-à- Assumptions: Hypotheses about factors or vis mandated roles and/or plans. This may risks which could affect the progress or suc- require a careful, even legally defensible, cess of a development intervention. demonstration that the work is consistent Note: Assumptions can also be understood with the contract terms. as hypothesized conditions that bear on the Note: Accountability in development may validity of the evaluation itself, e.g., about refer to the obligations of partners to act ac- the characteristics of the population when de- cording to clearly defined responsibilities, signing a sampling procedure for a survey. roles and performance expectations, often Assumptions are made explicit in theory- with respect to the prudent use of resources. based evaluations where evaluation tracks For evaluators, it connotes the responsibility systematically the anticipated results chain. to provide accurate, fair and credible moni- toring reports and performance assessments. Attribution: The ascription of a causal link be- For public sector managers and policy-mak- tween observed (or expected to be observed) ers, accountability is to taxpayers/citizens. changes and a specific intervention. Note: Attribution refers to that which is to Activity: Actions taken or work performed be credited for the observed changes or re- through which inputs, such as funds, techni- sults achieved. It represents the extent to cal assistance and other types of resources are which observed development effects can be mobilized to produce specific outputs. attributed to a specific intervention or to the Related term: development intervention. performance of one or more partner taking Analytical tools: Methods used to process and account of other interventions, (anticipated interpret information during an evaluation. or unanticipated) confounding factors, or ex- ternal shocks. Appraisal: An overall assessment of the rele- vance, feasibility and potential sustainability Audit: An independent, objective assurance of a development intervention prior to a deci- activity designed to add value and improve sion of funding. an organization's operations. It helps an or- Note: In development agencies, banks, etc., ganization accomplish its objectives by bring- the purpose of appraisal is to enable decision- ing a systematic, disciplined approach to as- makers to decide whether the activity repre- sess and improve the effectiveness of risk 223 224 Annex VI management, control and governance tion and analyses undertaken, through a processes. transparent chain of arguments. Note: a distinction is made between regu- Counterfactual: The situation or condition larity (financial) auditing, which focuses on which hypothetically may prevail for individ- compliance with the applicable statutes and uals, organizations, or groups where there is regulations; and performance auditing, which no development intervention. is concerned with relevance, economy, effi- ciency and effectiveness. Internal auditing Country Program Evaluation/ Country Assis- provides an assessment of internal controls tance Evaluation: Evaluation of one or more undertaken by a unit reporting to manage- donor's or agency's portfolio of development ment while external auditing is conducted by interventions, and the assistance strategy be- an independent organization. hind them, in a partner country. Base-line study: An analysis describing the Data collection tools: Methodologies used to situation prior to a development intervention, identify information sources and collect in- against which progress can be assessed or formation during an evaluation. comparisons made. Note: Examples are informal and formal surveys, direct and participatory observation, Benchmark: Reference point or standard community interviews, focus groups, expert against which performance or achievements opinion, case studies, literature search. can be assessed. Note: A benchmark refers to the perfor- Development intervention: An instrument for mance that has been achieved in the recent partner (donor and non-donor) support past by other comparable organizations, or aimed to promote development. what can be reasonably inferred to have been Note: Examples are policy advice, projects achieved in the circumstances. and programs. Beneficiaries: The individuals, groups, or or- Development objective: Intended impact con- ganizations, whether targeted or not, that tributing to physical, financial, institutional, benefit, directly or indirectly, from the devel- social, environmental, or other benefits to a opment intervention. society, community, or group of people via Related terms: reach, target groups. one or more development interventions. Cluster evaluation: An evaluation of a set of Economy: Absence of waste for a given output. related activities, projects and/or programs. Note: An activity is economical when the costs of the scarce resources used approxi- Conclusions: Conclusions point out the factors mate the minimum needed to achieve planned of success and failure of the evaluated inter- objectives. vention, with special attention paid to the in- tended and unintended results and impacts, Effect: Intended or unintended change due di- and more generally to any other strength or rectly or indirectly to an intervention. weakness. A conclusion draws on data collec- Related terms: results, outcome. Annex VI 225 Effectiveness: The extent to which the develop- Note: Evaluation in some instances in- ment intervention's objectives were achieved, volves the definition of appropriate stand- or are expected to be achieved, taking into ards, the examination of performance against account their relative importance. those standards, an assessment of actual and Note: Also used as an aggregate measure of expected results and the identification of rele- (or judgment about) the merit or worth of an vant lessons. activity, i.e., the extent to which an interven- Related term: review. tion has attained, or is expected to attain, its Ex-ante evaluation: An evaluation that is per- major relevant objectives efficiently in a sus- formed before implementation of a develop- tainable fashion and with a positive institu- ment intervention. tional development impact. Related terms: appraisal, quality at entry. Related term: efficacy. Ex-post evaluation: Evaluation of a develop- Efficiency: A measure of how economically re- ment intervention after it has been completed. sources/inputs (funds, expertise, time, etc.) Note: It may be undertaken directly after are converted to results. or long after completion. The intention is to Evaluability: Extent to which an activity or identify the factors of success or failure, to program can be evaluated in a reliable and assess the sustainability of results and im- credible fashion. pacts, and to draw conclusions that may in- Note: Evaluability assessment calls for the form other interventions. early review of a proposed activity in order to External evaluation: The evaluation of a devel- ascertain whether its objectives are ade- opment intervention conducted by entities quately defined and its results verifiable. and/or individuals outside the donor and im- Evaluation: The systematic and objective as- plementing organizations. sessment of an on-going or completed proj- Feedback: The transmission of findings gener- ect, program or policy, its design, implemen- ated through the evaluation process to parties tation and results. The aim is to determine for whom it is relevant and useful so as to fa- the relevance and fulfillment of objectives, cilitate learning. This may involve the collec- development efficiency, effectiveness, impact tion and dissemination of findings, conclu- and sustainability. An evaluation should pro- sions, recommendations and lessons from vide information that is credible and useful, experience. enabling the incorporation of lessons learned into the decision-making process of both re- Finding: A finding uses evidence from one or cipients and donors. more evaluations to allow for a factual Evaluation also refers to the process of de- statement. termining the worth or significance of an ac- Formative evaluation: Evaluation intended to tivity, policy or program. An assessment, as improve performance, most often conducted systematic and objective as possible, of a during the implementation phase of projects planned, on-going, or completed develop- or programs. ment intervention. 226 Annex VI Note: Formative evaluations may also be able use of its human, financial, and natural conducted for other reasons such as compli- resources, for example through: (a) better ance, legal requirements or as part of a larger definition, stability, transparency, enforceabil- evaluation initiative. ity and predictability of institutional arrange- Related term: process evaluation. ments and/or (b) better alignment of the mis- sion and capacity of an organization with its Goal: The higher-order objective to which a mandate, which derives from these institu- development intervention is intended to tional arrangements. Such impacts can in- contribute. clude intended and unintended effects of an Related term: development objective. action. Impacts: Positive and negative, primary and Internal evaluation: Evaluation of a develop- secondary, long-term effects produced by a ment intervention conducted by a unit and/or development intervention, directly or indi- individuals reporting to the management rectly, intended or unintended. of the donor, partner, or implementing Independent evaluation: An evaluation carried organization. out by entities and persons free of the control Related term: self-evaluation. of those responsible for design and implemen- Joint evaluation: An evaluation to which tation of the development intervention. different donor agencies and/or partners Note: The credibility of an evaluation de- participate. pends in part on how independently it has Note: There are various degrees of "joint- been carried out. Independence implies free- ness" depending on the extent to which indi- dom from political influence and organiza- vidual partners cooperate in the evaluation tional pressure. It is characterized by full ac- process, merge their evaluation resources and cess to information and by full autonomy in combine their evaluation reporting. Joint carrying out investigations and reporting evaluations can help overcome attribution findings. problems in assessing the effectiveness of Indicator: Quantitative or qualitative factor or programs and strategies, the complementarity variable that provides a simple and reliable of efforts supported by different partners, the means to measure achievement, to reflect the quality of aid coordination, etc. changes connected to an intervention, or to Lessons learned: Generalizations based on eval- help assess the performance of a development uation experiences with projects, programs, actor. or policies that abstract from the specific cir- Inputs: The financial, human, and material re- cumstances to broader situations. Frequently, sources used for the development intervention. lessons highlight strengths or weaknesses in preparation, design, and implementation that Institutional Development Impact: The ex- affect performance, outcome, and impact. tent to which an intervention improves or weakens the ability of a country or region to Logical framework (Logframe): Management make more efficient, equitable, and sustain- tool used to improve the design of interven- Annex VI 227 tions, most often at the project level. It in- Participatory evaluation: Evaluation method volves identifying strategic elements (inputs, in which representatives of agencies and outputs, outcomes, impact) and their causal stakeholders (including beneficiaries) work relationships, indicators, and the assumptions together in designing, carrying out and inter- or risks that may influence success and fail- preting an evaluation. ure. It thus facilitates planning, execution and Partners: The individuals and/or organizations evaluation of a development intervention. that collaborate to achieve mutually agreed Related term: results-based management. upon objectives. Meta-evaluation: The term is used for evalua- Note: The concept of partnership connotes tions designed to aggregate findings from a shared goals, common responsibility for out- series of evaluations. It can also be used to comes, distinct accountabilities and recipro- denote the evaluation of an evaluation to cal obligations. Partners may include govern- judge its quality and/or assess the perform- ments, civil society, non-governmental ance of the evaluators. organizations, universities, professional and business associations, multilateral organiza- Mid-term evaluation: Evaluation performed tions, private companies, etc. toward the middle of the period of implemen- tation of the intervention. Performance: The degree to which a develop- Related term: formative evaluation. ment intervention or a development partner operates according to specific criteria/stand- Monitoring: A continuing function that uses ards/guidelines or achieves results in accor- systematic collection of data on specified in- dance with stated goals or plans. dicators to provide management and the main stakeholders of an ongoing develop- Performance indicator: A variable that allows ment intervention with indications of the ex- the verification of changes in the develop- tent of progress and achievement of objectives ment intervention or shows results relative to and progress in the use of allocated funds. what was planned. Related term: performance monitoring, Related terms: performance monitoring, indicator. performance measurement. Outcome: The likely or achieved short-term Performance measurement: A system for and medium-term effects of an intervention's assessing performance of development inter- outputs. ventions against stated goals. Related terms: result, outputs, impacts, Related terms: performance monitoring, effect. indicator. Outputs: The products, capital goods and serv- Performance monitoring: A continuous ices that result from a development interven- process of collecting and analyzing data to tion; may also include changes resulting from compare how well a project, program, or the intervention which are relevant to the policy is being implemented against expected achievement of outcomes. results. 228 Annex VI Process evaluation: An evaluation of the inter- Note: examples of quality assurance activi- nal dynamics of implementing organizations, ties include appraisal, RBM, reviews during their policy instruments, their service delivery implementation, evaluations, etc. Quality as- mechanisms, their management practices, and surance may also refer to the assessment of the linkages among these. the quality of a portfolio and its development Related term: formative evaluation. effectiveness. Program evaluation: Evaluation of a set of Results-Based Management (RBM): A man- interventions, marshaled to attain specific agement strategy focusing on performance global, regional, country, or sector develop- and achievement of outputs, outcomes and ment objectives. impacts. Note: a development program is a time Related term: logical framework. bound intervention involving multiple activi- Review: An assessment of the performance ties that may cut across sectors, themes of an intervention, periodically or on an and/or geographic areas. ad hoc basis. Related term: Country program/strategy Note: Frequently "evaluation" is used for evaluation. a more comprehensive and/or more in-depth Project evaluation: Evaluation of an individual assessment than "review." Reviews tend to development intervention designed to achieve emphasize operational aspects. Sometimes specific objectives within specified resources the terms "review" and "evaluation" are and implementation schedules, often within used as synonyms. the framework of a broader program. Related term: evaluation. Note: Cost benefit analysis is a major in- Risk analysis: An analysis or an assessment of strument of project evaluation for projects factors (called assumptions in the logframe) with measurable benefits. When benefits can- that affect or are likely to affect the successful not be quantified, cost-effectiveness is a suit- achievement of an intervention's objectives. able approach. A detailed examination of the potential un- Project or program objective: The intended wanted and negative consequences to human physical, financial, institutional, social, envi- life, health, property, or the environment ronmental, or other development results to posed by development interventions; a sys- which a project or program is expected to tematic process to provide information re- contribute. garding such undesirable consequences; the process of quantification of the probabilities Purpose: The publicly stated objectives of the and expected impacts for identified risks. development program or project. Sector program evaluation: Evaluation of a Quality assurance: Quality assurance encom- cluster of development interventions in a sec- passes any activity that is concerned with as- tor within one country or across countries, sessing and improving the merit or the worth all of which contribute to the achievement of of a development intervention or its compli- a specific development goal. ance with given standards. Annex VI 229 Note: a sector includes development activi- Terms of reference: Written document present- ties commonly grouped together for the pur- ing the purpose and scope of the evaluation, pose of publication such as health, education, the methods to be used, the standard against agriculture, transport, etc. which performance is to be assessed or analy- ses are to be conducted, the resources and Self-evaluation: An evaluation by those who time allocated, and reporting requirements. are entrusted with the design and delivery of Two other expressions sometimes used with a development intervention. the same meaning are "scope of work" and Stakeholders: Agencies, organizations, groups "evaluation mandate." or individuals who have a direct or indirect Thematic evaluation: Evaluation of a selection interest in the development intervention or its of development interventions, all of which evaluation. address a specific development priority that Summative evaluation: A study conducted at cuts across countries, regions, and sectors. the end of an intervention (or a phase of that Triangulation: The use of three or more theo- intervention) to determine the extent to ries, sources or types of information, or types which anticipated outcomes were produced. of analysis to verify and substantiate an as- Summative evaluation is intended to provide sessment. information about the worth of the program. Note: by combining multiple data sources, Related term: impact evaluation. methods, analyses or theories, evaluators seek Sustainability: The continuation of benefits to overcome the bias that comes from single from a development intervention after major informants, single methods, single observer development assistance has been completed. or single theory studies. The probability of continued long-term Validity: The extent to which the data collec- benefits. The resilience to risk of the net tion strategies and instruments measure what benefit flows over time. they purport to measure. Target group: The specific individuals or organ- izations for whose benefit the development Source: OECD 2002a. intervention is undertaken. Notes 1. A complete list of the MDGs--including targets individual ministries, and development organiza- and indicators--can be found in annex 3. tions that wish to undertake a self-assessment is 2. "Technical cooperation expenditures totaled contained in annex 1, "Assessing Results-Based US$14.3 billion in 1999, according to the Develop- M&E Capacity: An Assessment for Countries, De- ment Assistance Committee (DAC) of the OECD. velopment Institutions and Their Partners." See This is a large amount, almost double the sum in also "A Diagnostic Guide and Action Framework" 1969. If personnel and training in investment and (Mackay 1999). Some questions in the readiness other projects are included, the figure would be assessment in this handbook are drawn from that even larger, $24.6 billion (Baris et al., 2002)" earlier work. (Fukuda-Parr, Lopes, and Malik 2002 pp. 3­4). 5. There are other models for setting good perform- 3. While we refer to the country as the unit of analy- ance indicators. For example, the UNDP uses an- sis here, we will immediately stress that the same other formula, the SMART principle; the charac- concept of a readiness assessment could be appli- teristics of good indicators are S=specific, cable to a sector, a region, a program, or even an M=measurable, A=attainable, R=relevant, and individual project. It is also applicable in civil so- T=trackable (Kahn 2001, p. 24). ciety and the private sector. 6. Also see annex 6 for a complete glossary of key 4. A version of the readiness assessment for countries, terms in evaluation and results-based management. 230 References Binnendijk, Annette. 2000. "Results Based Manage- velopment: Comparative Insights from Colombia, ment in the Development Cooperation Agencies: A China, and Indonesia," in Richard Boyle and Don- Review of Experience." Paper prepared for ald Lemaire, eds., Building Effective Evaluation OECD/DAC Working Party on Aid Evaluation. Capacity: Lessons from Practice. New Brunswick, Paris. February 10­11. (Revised October 2000.) N.J.: Transaction Publishers. Carroll, Lewis. 1865. Alice's Adventures in Wonder- Hatry, Harry P. 1999. Performance Measurement: land. Reprint edition 2002. New York: Sea Star Getting Results. Washington, D.C.: The Urban In- Books. stitute Press. ChannahSorah, Vijaya Vinita. 2003. "Moving from Hatry, Harry P., Elaine Morley, Shelli B. Rossman, Measuring Processes to Outcomes: Lessons Joseph P. Wholey. 2003. "How Federal Programs Learned from GPRA in the United States." Pre- Use Outcome Information: Opportunities for Fed- sented at World Bank and Korea Development In- eral Managers." Washington, D.C.: IBM Endow- stitute joint conference on Performance Evaluation ment for The Business of Government. System and Guidelines with Application to Large- Hauge, Arild. 2001. "Strengthening Capacity for Scale Construction, R&D, and Job Training In- Monitoring and Evaluation in Uganda: A Results vestments. Seoul, South Korea. July 24­25. Based Perspective." World Bank Operations Evalu- Crawford, David. 2003. "With Help from Corpora- ation Dept. ECD Working Paper Series, Number tions, German Group Fights Corruption." Wall 8. Washington, D.C. Street Journal, November 26. IDA (International Development Association). 2002. The Daily Star. 2003. "Saidi Predicts Gains from "Measuring Outputs and Outcomes in IDA Coun- Joining IMF Data System: Part of Good Govern- tries." IDA 13. World Bank. Washington, D.C. ance." January 28. IFAD (International Fund for Agricultural Develop- Dorotinsky, William. 2003a. "Active and Passive Ap- ment). 2002. "A Guide for Project M&E: Manag- proaches to Use of Results Findings." World Bank. ing for Impact in Rural Development." Rome: Personal communication with authors, December IFAD. Available at http://www.ifad.org/evalua- 5, 2003. tion/guide/ Dorotinsky, William. 2003b. Information on Moni- IMF (International Monetary Fund). 2002. "What is toring for Results in Brazil. World Bank. Personal the General Data Dissemination System (GDDS)?" communication with authors, December 5, 2003. Washington, D.C.: IMF. Fukuda-Parr, Sakiko, Carlos Lopes, and Khalid ------. 2003. "Financial Soundness Indicators." Malik, eds. 2002. Capacity for Development: Washington, D.C.: IMF. Available at New Solutions to Old Problems. London: Earth- http://imf.org/external/np/sta/fsi/eng/fsi.htm scan Publications, Ltd. Khan, M. Adil. 2001. A Guidebook on Results Based Furubo, Jan-Eric, Ray C. Rist, and Rolf Sandahl, eds. Monitoring and Evaluation: Key Concepts, Issues 2002. International Atlas of Evaluation. New and Applications. Monitoring and Progress Re- Brunswick, N.J.: Transaction Publishers. view Division, Ministry of Plan Implementation, Guerrero, R. Pablo. 1999. "Evaluation Capacity De- Government of Sri Lanka. Colombo, Sri Lanka. 231 232 References Kumar, Krishna, ed. 1993. Rapid Appraisal Methods. Development). 2001. "Evaluation Feedback for World Bank. Washington, D.C. Effective Learning and Accountability." Paris: Kusek, J. Z. and R. C. Rist. 2001. "Building a Perfor- OECD/ DAC. mance-Based Monitoring and Evaluation System: ------. 2002a. "Glossary of Key Terms in Evaluation The Challenges Facing Developing Countries." and Results-Based Management." Paris: OECD/ Evaluation Journal of Australasia. 1(2): 14­23. DAC. ------. 2003. "Readiness Assessment: Toward Per- ------. 2002b. Public Management and Governance formance Monitoring and Evaluation in the Kyr- (PUMA). "Overview of Results-Focused Manage- gyz Republic." Japanese Journal of Evaluation ment and Budgeting in OECD Member Coun- Studies. 3(1): 17­31. tries." Twenty-third Annual Meeting of OECD Se- Lee, Yoon-Shik. 1999. In Richard Boyle and Donald nior Budget Officials. Washington, D.C. June 3­4. Lemaire, eds. Building Effective Evaluation Ca- Osborne, David and Ted Gaebler. 1992. Reinventing pacity: Lessons from Practice. New Brunswick, Government. Boston, Mass.: Addison-Wesley N.J.: Transaction Publishers. Publishing. ------. 2002. In Jan-Eric Furubo, Ray C. Rist, and Picciotto, Robert. 2002. "Development Cooperation Rolf Sandahl, eds. International Atlas of Evalua- and Performance Evaluation: The Monterrey tion. New Brunswick, N.J.: Transaction Publishers. Challenge." World Bank working paper prepared Leeuw, Frans L. 2003. "Evaluation of Development for roundtable on "Better Measuring, Monitoring, Agencies' Performance: The Role of Meta-Evalua- and Managing for Development Results," spon- tions." Paper prepared for the Fifth Biennial World sored by the Multilateral Development Banks in Bank Conference on Evaluation and Development. cooperation with the Development Assistance Washington, D.C. July 15­16. Committee of the Organisation for Economic Co- Mackay, Keith. 1999. "Evaluation Capacity Develop- operation and Development. Washington, D.C., ment: A Diagnostic Guide and Action Frame- June 5­6. work." World Bank Operations Evaluation De- President of the Treasury Board of Canada. 2002. partment. ECD Working Paper Series, Number 6. "Canada's Performance 2002: Annual Report to Washington, D.C. Parliament." Ottawa, Canada. ------. 2002. "The Australian Government: Success Republique Française. 2001. Ministère de l'Economie with a Central, Directive Approach," in Furubo, des Finances et de l'Industrie. "Towards New Pub- Rist, and Sandahl, eds., International Atlas of lic Management. Newsletter on the Public Finance Evaluation. New Brunswick, N.J.: Transaction Reform." No. 1. September. Publishers. Schacter, Mark. 2000. "Sub-Saharan Africa: Lessons Marchant, Tim. 2000. Africa Region presentation. from Experience in Supporting Sound Govern- World Bank. Washington D.C. ance." World Bank Operations Evaluation Depart- NYC.gov. 2003. "New York City Continues to be the ment. ECD Working Paper Series, Number 7. Nation's Safest Large City." http://www.nyc.gov/ Washington, D.C. html/om/html/2003a/crime_falls.html Schiavo-Campo, Salvatore. 1999. "`Performance' in O'Connell, Paul E. 2001."Using Performance Data the Public Sector." Asian Journal of Political Sci- for Accountability: The New York City Police De- ence 7(2): 75­87. partment's CompStat Model of Police Manage- Sri Lanka Evaluation Association and the Ministry of ment." PricewaterhouseCoopers Endowment for Policy Development and Implementation. 2003. The Business of Government: Arlington, Va. "National Evaluation Policy for Sri Lanka." OECD (Organisation for Economic Co-operation and Colombo, Sri Lanka. References 233 Stiglitz, Joseph and Roumeen Islam. 2003. "Informa- U.S. Office of Management and Budget. 1993. "Gov- tion is Capital." Le Monde. January 3. ernment Performance Results Act of 1993." Wash- TI (Transparency International). 1997. Available at ington, D.C. http://www.transparency.org/ United Way of America. 1996. "Outcome Measure- ------. 2002. Available at http://www.transparency.org/ ment Resource Network." Available at http://na- Treasury Board Secretariat of Canada. 2001. "Guide tional.unitedway.org/outcomes/resources/mpo/con- for the Development of Results-Based Management tents.cfm and Accountability Frameworks." Ottawa, Canada. Valadez, Joseph, and Michael Bamberger. 1994. Mon- Tufte, Edward R. 2001. The Visual Display of Quanti- itoring and Evaluating Social Programs in Devel- tative Information. Cheshire, Conn.: Graphics Press. oping Countries: A Handbook for Policymakers, ------. 2002. Visual Explanations: Images and Managers, and Researchers. Washington, D.C.: Quantities, Evidence and Narrative. Cheshire, The World Bank. Conn.: Graphics Press. Webb, E. J., D. T. Campbell, R. D. Schwartz, U.K. Cabinet Office. n.d. Available at L. Sechrest, 1966. Unobtrusive Measures: Nonre- http://www.cabinet-office.gov.uk active Research in the Social Sciences. Chicago: United Nations. n.d. Available at Rand McNally. http://www.un.org/millenniumgoals/ Wholey, Joseph S. 2001. "Managing for Results: ------. 2003. "Indicators for Monitoring the Millen- Roles for Evaluators in a New Management Era." nium Development Goals." New York, N.Y.: The American Journal of Evaluation. 22(3): United Nations. Available at http://www.develop- 343­347. mentgoals.org/mdgun/MDG_metadata_08-01- Wholey, Joseph S., Harry Hatry, and Kathryn New- 03_UN.htm comer. 1994. Handbook of Practical Program UNDP (United Nations Development Programme). Evaluation. San Francisco: Jossey-Bass Publishers. 2001. "Human Development Report." New York: World Bank. "Core Welfare Indicators Question- Oxford University Press. Also available at naire." Washington, D.C. Available at http://www.undp.org/hdr2001/faqs.html##1 http://www4.worldbank.org/afr/stats/pdf/cwiq.pdf ------. 2002. Handbook on Monitoring and Evaluat- ------. "Ghana Core Welfare Indicators." Washing- ing for Results. New York: UNDP Evaluation Office. ton, D.C. Available at http:// UNPF (United Nations Population Fund). 2002. www.worldbank.org/afr/stats/pdf/ghcoreinds.pdf "Monitoring and Evaluation Toolkit for Pro- ------. "Introducing the Core Welfare Indicators gramme Managers." Office of Oversight and Eval- Questionnaire (CWIQ)." Washington, D.C. Avail- uation. Available at http://www.unfpa.org/moni- able at http://www4.worldbank.org/afr/stats/ toring/toolkit.htm pdf/cwiqloop.pdf U.S. Department of Labor. 2002. "Annual Perfor- ------. 1997. World Development Report 1997: The mance Plan: Fiscal Year 2003." Washington, D.C. State in a Changing World. New York: Oxford U.S. General Accounting Office (GAO). 2002. "High- University Press. lights of a GAO Forum, Mergers and Transforma- ------. 2000. "Rural Development Indicators Hand- tion: Lessons Learned for a Department of Home- book." 2nd Edition. Washington, D.C. Available at land Security and Other Federal Agencies." http://www-wds.worldbank.org/servlet/ Washington, D.C. WDS_IBank_Servlet?pcont=details&eid=0000949 ------. 2003. "Program Evaluation: An Evaluation 46_01061604041624 Culture and Collaborative Partnerships Help Build ------. 2001a. International Program for Develop- Agency Capacity." Washington, D.C. ment Evaluation Training (IPDET). 234 References http://www.worldbank.org/oed/ipdet/ ------. 2002d. "Republic of Albania: Establishment ------. 2001b. "Outcomes-Based Budgeting Systems: of a Permanent System of Household Surveys for Experience from Developed and Developing Poverty Monitoring and Policy Evaluation." Countries." Washington, D.C. Concept document. Washington, D.C. ------. 2001c. "Readiness Assessment--Toward ------. 2003a. Operations Evaluation Department Results-Based Monitoring and Evaluation in Web site: http://www.worldbank.org/oed/ecd/. Egypt." Washington, D.C. ------. 2003b. Overview of Poverty Reduction ------. 2001d. "Readiness Assessment--Toward Strategies. http://www.worldbank.org/poverty/ Results-Based Monitoring and Evaluation in Ro- strategies/overview.htm mania." Washington, D.C. ------. 2003c. "Toward Country-Led Development: ------. 2001e. "Readiness Assessment--Toward A Multi-Partner Evaluation of the CDF." OED Results-Based Monitoring and Evaluation in the Precis, Number 233. Washington, D.C. Philippines." Washington, D.C. Worthen, Blaine, James Sanders, and Jody Fitz- ------. 2002a. Albania country documents and patrick. 1997. Program Evaluation: Alternative reports. Washington, D.C. Available at Approaches and Practical Guidelines. New York: http://lnweb18.worldbank.org/eca/albania.nsf Longman Publishers. ------. 2002b. "Heavily Indebted Poor Country Wye, Chris. 2002. "Performance Management: A Initiative (HIPC)." Washington, D.C. Available at `Start Where You Are, Use What You Have' http://www.worldbank.org/ug/hipc.htm Guide." Arlington, Va.: IBM Endowment for Busi- ------. 2002c. "Readiness Assessment--Toward ness in Government. Managing for Results Series. Results-Based Monitoring and Evaluation in Bangladesh." Washington, D.C. Useful Web Sites E-Government in New Zealand: http://www.e-gov- Transparency International: http://www.transparency. ernment.govt.nz/ org/ Egyptian Museum, Cairo: United Nations Online Network in Public Adminis- http://www.touregypt.net/egyptmuseum/egyptian_ tration and Finance museum.htm http://www.unpan.org/ Monitoring and Evaluation Capacity Development: USAID Center for Development Information and http://www.worldbank.org/oed/ecd/ Evaluation: http://www.usaid.gov/pubs/usaid_eval/ Monitoring and Evaluation News: World Bank Operations Evaluation Department: http://www.mande.co.uk/ http://www.worldbank.org/oed/ OECD Development Assistance Committee Working www.worldbank.org/oed/ecd (M&E capacity Party on Aid Evaluation: building) http://www.oecd.org/home/ http://www.worldbank.org/oed/ipdet/ OECD E-Government: WWW Virtual Library: Evaluation, http://www. http://www.oecd.org/department/0,2688,en_2649_ policy-evaluation.org/ 34131_1_1_1_1_1,00.html OECD Public Management Program: http://www1.oecd.org/puma/ 235 Additional Reading Introduction ------. 2002. "Building Results-Based Monitoring and Evaluation Systems: Assessing Developing Kettl, Donald F. 2002. The Global Public Manage- Countries' Readiness." Zeitschrift für Evaluation, ment Revolution: A Report on the Transformation (1): 151­158. of Governance. Washington, D.C.: Brookings ------. 2003. "Readiness Assessment: Toward Institution. Performance Monitoring and Evaluation in the Kusek, J. Z. and R. C. Rist. 2000. "Making M&E Kyrgyz Republic." Japanese Journal of Evaluation Matter--Get the Foundation Right." Evaluation Studies. 3(1):17­31. Insights. 2(2):7­10. ------. 2002. "Building Results-Based Monitoring Chapter 2 and Evaluation Systems: Assessing Developing International Fund for Agricultural Development Countries Readiness." Zeitschrift für Evaluation. (IFAD). 2002. "Managing for Impact in Rural (1): 151­158. Development: A Guide for Project M&E." Rome: ------. 2003. "Readiness Assessment: Toward IFAD. Available at http://www.ifad.org/evalua- Performance Monitoring and Evaluation in the tion/guide/ Kyrgyz Republic." Japanese Journal of Evaluation Khan, M. Adil. 2001. "A Guidebook on Results Studies. 3(1): 17­31. Based Monitoring and Evaluation: Key Concepts, Mayne, John and Eduardo Zapico-Goni, eds. 1999. Issues and Applications." Monitoring and Progress Monitoring Performance in the Public Sector: Review Division, Ministry of Plan Implementa- Future Directions from International Experience. tion, Government of Sri Lanka. Colombo, Sri New Brunswick, N.J.: Transaction Publishers. Lanka. Picciotto, Robert and Eduardo D. Wiesner, eds. 1998. International Development Association (IDA). 2002. Evaluation and Development. New Brunswick, "Measuring Outputs and Outcomes in IDA Coun- N.J.: Transaction Publishers. tries." IDA 13. World Bank, Washington, D.C. The World Bank. 1994. Report of the Evaluation Capacity Task Force. Washington, D.C. Chapter 3 Chapter 1 Hatry, Harry P. 2001. "What Types of Performance Information Should be Tracked," in Dall W. Boyle, Richard and Donald Lemaire, eds. 1999. Forsythe, ed., Quicker, Better, Cheaper? Managing Building Effective Evaluation Capacity: Lessons Performance in American Government. Albany, from Practice. New Brunswick, N.J.: Transaction N.Y.: Rockefeller Institute Press. Publishers. Kusek, J. Z. and Ray C. Rist. 2000. "Making M&E Kusek, Jody Zall, and Ray C. Rist. 2000. "Making Matter--Get the Foundation Right." Evaluation M&E Matter--Get the Foundation Right." Evalu- Insights 2(2):7­10. ation Insights. 2(2):7­10 Mayne, John and Eduardo Zapico-Goni, eds. 1999. ------. 2001. "Building a Performance-Based M&E Monitoring Performance in the Public Sector: Fu- System: The Challenges Facing Developing Coun- ture Directions from International Experience. tries." Evaluation Journal of Australasia. New Brunswick, N.J.: Transaction Publishers. 1(2):14­23. 236 Additional Readings 237 Shand, David. 1998. "The Role of Performance Indi- Rist, Ray C., ed. 1999. Program Evaluation and the cators in Public Expenditure Management." IMF Management of Government. New Brunswick, Working Paper. Washington, D.C.: International N.J.: Transaction Publishers. Monetary Fund. Vedung, Evert. 1997. Public Policy and Program U.S. Department of Justice. National Institute of Evaluation. New Brunswick, N.J.: Transaction Justice. "Mapping and Analysis for Public Safety." Publishers. Washington, D.C.: U.S. Department of Justice. Wholey, J. S., H. P. Hatry, and K. E. Newcomer, eds. Available at http://www.ojp.usdoj.gov/nij/maps/ 1994. Handbook of Practical Program Evalua- U.S. Department of the Treasury, Financial Management tion. San Francisco, Calif.: Jossey-Bass Publishers. Service. 1993. "Performance Measurement Guide." Worthen, B. R., J. R. Sanders, and J. L. Fitzpatrick. Washington, D.C.: US Department of the Treasury. 1997. Program Evaluation: Alternative Ap- Wye, Chris. 2002. "Performance Management: A proaches and Practical Guidelines, 2nd ed. New `Start Where You Are, Use What You Have' York, N.Y.: Addison, Wesley, and Longman. Guide." Arlington, Va.: IBM Endowment for Busi- Chapter 8 ness in Government. Managing for Results Series. Creswell, John W. 1994. Research Design: Qualita- Chapter 5 tive and Quantitative Approaches. Thousand Wye, Chris. 2002. "Performance Management: A Oaks, Calif.: Sage Publications. `Start Where You Are, Use What You Have' Kumar, Krishna, ed. 1993. Rapid Appraisal Methods. Guide." Arlington, Va.: IBM Endowment for Busi- World Bank. Washington, D.C. ness in Government. Managing for Results Series. Rist, Ray C. 1994. "Influencing the Policy Process with Qualitative Research," in Norman K. Denzin Chapter 6 and Yvonna S. Lincoln, eds. Handbook of Quali- Mayne, John and Eduardo Zapico-Goni, eds. 1999. tative Research. Thousand Oaks, Calif.: Sage Monitoring Performance in the Public Sector: Fu- Publications. ture Directions from International Experience. World Bank. 2003. International Program for New Brunswick, N.J.: Transaction Publishers. Development Evaluation Training. Available at United National Development Programme (UNDP). http://www.worldbank.org/oed/ipdet/ and 2002. Handbook on Monitoring and Evaluating http://www.carleton.ca/ipdet/ for Results. New York: UNDP Evaluation Office. Chapter 9 Chapter 7 Leeuw, Frans L., Ray C. Rist, and Richard C. Creswell, John W. 1994. Research Design: Qualita- Sonnichsen. 1994. Can Governments Learn? tive and Quantitative Approaches. Thousand Comparative Perspectives on Evaluation and Oaks, Calif.: Sage Publications. Organizational Learning. New Brunswick, N.J.: Furubo, J. E., R. C. Rist, and R. Sandahl, eds. 2002. Transaction Publishers. International Atlas of Evaluation. New Rist, Ray C. 1997. "Evaluation and organizational Brunswick, N.J.: Transaction Press. learning." Evaluation Journal of Australasia. French Council for Evaluation. 1999. A Practical 9(1&2). Guide to Program and Policy Evaluation. Paris, Chapter 10 France: Scientific and National Councils for Evaluation. Boyle, Richard and Donald Lemaire, eds. 1999. Patton, Michael Q. 2002. Qualitative Research and Building Effective Evaluation Capacity: Lessons Evaluation Methods, 3rd ed. Thousand Oaks, from Practice. New Brunswick, N.J.: Transaction Calif.: Sage Publications. Publishers. 238 Additional Readings Georghiou, Luke. 1995. "Assessing the Framework Future Directions from International Experience. Programmes." Evaluation. 1(2): 171­188. New Brunswick, N.J.: Transaction Publishers. Ittner, Christopher D., and David F. Larcker. 2003. Pollitt, Christopher. 1995. "Justification by Works or "Coming Up Short on Nonfinancial Performance by Faith?" Evaluation. I(2): 133­154. Measurement." Harvard Business Review. 81(11): ------. 1997. "Evaluation and the New Public 88­95. Management: An International Perspective." Eval- Mayne, John and Eduardo Zapico-Goni, eds. 1999. uation Journal of Australasia. 9(1&2): 7­15. Monitoring Performance in the Public Sector: Index A B accountability, 9, 10, 17, 130, 160 Bangladesh, 33, 34, 154, 165 Brazil, 20, 102b6.2 government capacity for M&E, 54 and budget reforms in Malaysia, 36bi.ix and readiness assessment, 49, 50b1.1 culture of, 34, 145b9.5 Bangladesh Bureau of Statistics, 50b1.1 definition, 2 bar chart, 137f8.2 demand for, 44, 139b9.1 baselines, 9, 24, 77, 167, 168 demonstration of, 37, 140 building of, 82t4.1, 82­83, 167 and e-administration in Romania, 52b1.3 developing countries, 33, 88b4.1, 89 executive, 141b9.2 for an education policy, 81f4.2 German aid agencies, 143, 144b9.4 and indicators, 81­82, 167 and GRPA, 156b10.2 measurements for, 75 and engaging civil society and citizen groups, 148 and outcomes, 60 manager's role in, 139 overview, 80­81 Mexico, 101b6.1 and pretesting, 112 politics and, 45 and readiness assessments, 46 promotion and provision of, 21, 26, 46, 163 and reporting of data, 22, 132­33 and resource allocation, 100­101 and targets, 91­92, 93, 94, 95f5.3 and sustainability of M&E systems, 153­54, 170 and trends, 132 and Transparency International, 5, 6bi.iii U.S. Department of Labor, 142b9.3 activity-based management system, 98 see also information activity, definition, 223 base-line study advocates. see champions definition of, 224 Africa, Core Welfare Indicators, 76b3.2 benchmarks, 57, 102b6.2 African Development Bank, 32 definition of, 224 aid agencies, and evaluation-based learning, 143, 144b9.4 beneficiaries aid financing, 3 definition of, 224 Albania, 6, 26, 78b3.4, 88b4.1, 89 Better Government Initiative, United Kingdom, 155b10.1 Albanian Institute of Statistics (INSTAT), 88b4.1, 89 Bolivia, 70 analytical tools Brazil, 35, 100, 102b6.2 definition of, 223 Bribe Payers Index, 6bi.iii Andhra Pradesh (India) Performance Accountability Act budget process, 19, 28, 100 (Annex V), 211­222 Brazil, 102b6.2 appraisals France, 30bi.vii definition of, 223 Indonesia, 35 assumptions Malaysia, 35, 36bi.ix definition of, 223 Mexico, 101b6.1 attribution, 14, 72, 113, 114, 125 and OECD countries, 163­64 definition of, 223 publication of annual budget reports, 148 audiences Romania, 52b1.3 engagement of, 148­49 Uganda, 37bi.x and reporting of findings, 130­32, 133t8.1, 169 U.S. Department of Labor, 142b9.3 understanding of, 146 Auditor General, Office of, 149­50 C audits and auditing, 28, 31bi.vii, 149­50, 223­24 Canada, 27, 35, 149b9.8, 149­50 Australia, 27, 28, 29bi.vi, 35, 139­40 239 240 Index capacity compliance, 15 Albania, 88b4.1, 89 Comprehensive Development Framework (CDF), 9 assessment of, 45­46 CompStat, 141b9.2 Bangladesh, 50b1.1 Conclusions for data collection, 84, 88b4.1, 89 definition of, 224 development and institutionalization of, 154, 157t10.1 consensus building, 58, 116 in Egypt, 51b1.2 consultative process, in choosing outcomes, 58 of government, 174 Core Welfare Indicators Questionnaire (CWIQ), 76b3.2 and M&E systems, 47­48, 154, 170, 174 corruption, Bangladesh, 50b1.1 in workforce, 33 Corruption Perception Index, 6bi.iii capacity building Costa Rica, 34 Albania, 88b4.1 Counterfactual, 125 and M&E systems, 21­22, 42­43, 54­55, 177 definition of, 224 and readiness assessments, 46 CREAM (see indicators and performance indicators) case studies country program evaluation Bangladesh, 50b1.1 definition of, 224 Egypt, 51b1.2 credibility as evaluation type, 121f7.4 of information, 153 overview, 124­25, 169 of monitoring systems, 107f6.8, 108, 168 causal model, 122 crime, use of performance data to track, 141b9.2 CDF. see Comprehensive Development Framework (CDF) customer groups, 26 champions, 165 CREAM. see indicators and performance indicators Bangladesh, 50b1.1 CWIQ. see Core Welfare Indicators Questionnaire (CWIQ) Egypt, 51b1.2 identification of, 41­42, 44­45, 46 D need for, 53 data reaction to negative nature of M&E information, 46­47 analyzing and reporting of, 111­12 Romania, 52b1.3 credibility of, 108, 168 and turnover of personnel, 53 data dumps, 131 Charter Quality Networks, United Kingdom, 155b10.1 Egypt, 51b1.2 charts, 134­136, 137f8.2 key criteria of, 108­10 Chicago Museum of Science and Industry, 71b3.1 ownership of, 106­7 child morbidity, 101, 104f6.6 presentation of, 131, 132­36, 137f8.2 child mortality, 200 pretesting of, 112, 168 Chile, 35 reliability of, 109, 109f6.9, 109f6.10, 112 China, 34, 154, 157t10.1 sources of, 83­84 citizen groups, engagement of, 148­49 timeliness of, 109, 109f6.9, 109f6.10, 112 Citizen's Charters, United Kingdom, 154, 155b10.1 uses of, 42, 45­46, 141b9.2 civil servants, as partners in business, 139 validity of, 109, 109f6.9, 109f6.10, 112 civil society, 39, 52b1.3, 148­49, 162 see also information cluster bar chart, 137f8.2 data collection, 33, 46, 153, 154, 167 cluster evaluation Bangladesh, 50b1.1 definition of, 224 capacity in Albania, 88b4.1, 89 Colombia, 154, 157t10.1 continuous system of, 152­53 combination chart, 137f8.2 and CWIQ, 76b3.2 commercialization, 10, 162 designing and comparing methods of, 84­86, 87t4.2, common goals, 108, 166 167 communication in developing countries, 88b4.1, 89 and presentation of findings, 130­31 and indicators, 66, 70, 75, 167 line of sight, 48, 108, 139, 158 Lebanon, 89b4.2 strategy for dissemination of information, 146­50, management and maintenance of, 107­8 169­70 methods, 84­87t4.2 within and between government agencies, 165 pretesting of instruments and procedures for, 112, 168 Index 241 and rapid appraisal, 123­26 how not to construct outcome statements in, 63f2.4 Romania, 52b1.3 indicators for, 67, 68f3.2, 74 systems for, 87 outcome statements in a policy area, 62f2.3 tools for, 224 primary, 200 value for money, 127, 169 setting targets for, 93 data quality triangle, 109f6.10, 110f6.11, f6.12 effect decentralization, 10 definition of, 224 deregulation, 10, 162 effectiveness, 12, 163 decisionmaking definition, 225 and data presentation, 134­36 and evaluation, 15 and evaluation information, 116, 168 improvements in, 101 and feedback, 46 perceptions of, 21 and findings, 29bi.vi, 140, 169 of poverty reduction strategy, 37bi.x demand for monitoring and evaluation systems, 41­44, 49, of service delivery, 37bi.x 53, 152, 170 U.S. Department of Labor, 142b9.3 Department of Labor, U.S., results-based M&E system, and U.S. Government Results and Performance Act, 142b9.3 156b10.2 developed countries and use of resources, 162 achieving MDGs, 4 efficiency, 12, 15 experience in M&E, 2, 27­28 definition, 225 developing countries in governmental operations, 36bi.ix becoming part of global community, 3­4 improvements in, 101 data collection in, 88b4.1, 89 of public sector management, 31bi.viii experience in M&E systems, 32­34, 35­38, 164 of service delivery, 16­17 overview of readiness assessment in, 48­49 of U.S. federal programs, 156b10.2 setting indicators in, 75­79 and use of information, 88b4.1 development e-government, Jordan, 148 global partnership for, 202 Egypt, 11, 22 targets related to, 93­94, 94b5.1 capacity of, 54 Development Assistance Committee, OECD, 230n2 champions for advocating for M&E systems, 33 development goals, 55, 164 pilot program testing, 26 achieving of, 41, 105­6, 106f6.7 and readiness assessment, 51b1.2, 53, Annex II, range of, 21 178­199 and readiness assessment, 42 enclave-based approach to M&E, 24­25 tracking of, 72 environment sustainability, 201 development intervention, 12, 224 European Union (EU), 3, 7­8, 44, 57 development objectives, 76b3.2, 105, 224 European Union accession countries, and feedback disaggregation of outcomes, 59­60, 67 systems, 44 disclosure of performance findings, 147 European Union Structural Funds, 3, 8 disincentives, in sustaining monitoring and evaluation sys- evaluability, 225 tems, 154, 155, 158b10.4 evaluation, 24 donors capacity development and institutionalization of, 154, and choosing outcomes, 58 157t10.1 and development of M&E systems, 37­38 as complement to monitoring, 13­14 resources for IDA funding, 7 characteristics of quality, 126­127, 126f7.5 and technical assistance and training, 22, 33­34, 230n2 culture of , 160b10.5 collaborative partnerships and, 160b10.5 E definition, 12, 15, 225 e-administration, Romania, 52b1.3 examples of, 128, 128f7.6 economy and issues to consider in choosing outcomes, 57­58 definition of, 224 levels of, 13­14 education overview, 113­15 developing baseline data for a policy on, 81f4.2 quality and trustworthiness of, 126­28, 169 242 Index and rapid appraisal, 121f7.4, 123­26, 169 definition, 225 evaluation (continued) dissemination of, 127, 147, 169 relationship to monitoring, 12­15, 114 incentives for use of, 146b9.6 roles of, 13­15 integration of, 77b3.3 technical adequacy of, 127, 129 negative news, 136, 146­47 timing of, 118­21, 169 overview, 129 types of, 121­23, 169 presentation of, 131, 132­36, 137f8.2 uses of, 115­20, 168 presenting negative news, 136, 146­47 evaluation architecture, 4 and rapid appraisals, 124 evaluation-based learning, German aid agencies, 143, 144b9.4 sharing and comparing of, 150 evaluation culture, adoption of, 29, 32 trustworthiness of, 160b10.5 evaluation training, Egypt, 51b1.2 uses of, 111­12, 130­32, 138­40, 154, 169 ex-ante evaluation follow-up, 146 definition of, 225 foreign investment, 44 executive summaries, 134 formative evaluation expenditure framework, 16 definition of, 225­26 expenditures, 28, 34­35 France, 27, 28, 139 ex-post evaluation government reform in, 30bi.vii definition of, 225 freedom of information, 148 external application of monitoring and evaluations systems, funding, levels of, 92­93 19­20 external evaluations, 22, 225 G external pressures, and evaluation issues, 27­28 Gant chart, 97f6.2, 97 GAO. see U.S. General Accounting Office F gender equality and MDGs, 200 farmers' markets, 79b3.6 General Data Dissemination System (GDDS), IMF, 89b4.2 feedback, 12, 129, 166 Geographic Information Systems, 88b4.1 Albania, 78b3.4 German aid agencies, and evaluation-based learning, 143, benefits of, 140, 143­44 144b9.4 and decisionmaking, 46 Germany, 27 definition, 225 Giuliani, Rudolph, 141b9.2 disruption in loops of, 107 glossary, terms used in evaluation and results-based manage- and dissemination, 126­127 ment, 223­29 and evaluations, 126f7.5, 127, 169 goals, 7, 9, 35, 58, 94b5.1 flow of, 143, 150, 167 achieving of, 11, 12, 46, 139, 165, 167 German aid agencies, 143, 144b9.4 clarification of, 19 and incentives, 158b10.3 definition, 226 and indicators, 24, 66, 75 disaggregation of, 59­60, 74 for international organizations, 44 and feedback, 15, 66 and learning process, 143 gender-related, 26 as management tool, 130­31, 132, 139 of MDBs, 8 and oral presentations, 134 MDGs, 3, 4bi.i, 5, 5bi.ii, 72, 73, 92­93, 200­203 and progress of development activities, 140 M&E systems links to, 48 providing of, 15, 19, 20, 22, 34­35, 65 and partnerships, 105­6, 106f6.7 and rapid appraisal, 123 and rapid appraisal, 123­26 system, 144b9.4 setting of, 56, 58, 166 uses of, 138 U.S. Department of Labor, 142b9.3 Financial Soundness Indicators, IMF, 73 vs. outcomes, 56­57 findings, 24, 169 Gore, Al, 147b9.7 audiences for, 130­32, 133t8.1, 169 governance, 1, 7, 21 benefits of using, 140­44, 145b9.5, 146b9.6 government cross-study, 125 and building of evaluation culture and partnerships, and decisionmaking, 29bi.vi 160b10.5 Index 243 capacity to design M&E systems, 174 (Annex V), 211­222 changes in size and resources of, 10 indicators, 7, 24, 58, 98, 133t8.1 communication between and among, 165 ambiguity of performance and, 119, 120 reform in, 28, 30bi.vii and baseline information, 81­83, 167 roles and responsibilities for assessing performance of, checklist for assessing, 70, 71f3.3 53­54, 176­77 construction of, 60, 74­75, 166­67 stimulating cultural change in, 160­61, 160b10.5 cost of setting, 70, 87­88 turnover among officials, 53 CREAM, 66, 166, see also performance indicators United Kingdom, 155b10.1 data collection system for, 81­82, 87­88, 109, U.S. Government Performance and Results Act, 154, 109f6.9, 109f6.10, 167 156b10.2 definition, 65, 226 United States, 142b9.3 dilemmas, 71b3.1 graphs, 134, 137f8.2 experience in developing countries, 75­79 Growth and Poverty Reduction Strategy, Albania, 88b4.1, 89 identifying data sources for, 83­84 U.S. Government Performance and Results Act (GPRA) of Labor Department, 142b9.3 1993, 142b9.3, 154, 156b10.2 MDGs, 200­203 measurement of, 57, 109­10, 118, 169 H monitoring of, 101b6.1 HDI. see Human Development Index (HDI) and outcomes, 57, 79b3.6 Highly Indebted Poor Country (HIPC) Initiative, 3, 5­6, 9, piloting of, 86­89 37bi.ix predesigned, 72­74 HIV/AIDS, 117, 201 and presentation of data, 133 horizontal learning, 144b9.4 program and project level, 79b3.5 horizontal sharing of information, 104­5, 168 proxy, 70­72, 166 household surveys, 71­72 PRSPs, 8­9 human development, measures of, 72­73 setting of, 57, 63, 64, 66 Human Development Index (HDI), 72­73 and targets, 3, 5bi.ii, 91, 95f5.3 human resources, 159 tracking of, 85 translating outcomes into, 66­67, 68f3.2 I see also performance indicators IDA. see International Development Association (IDA) indirect indicators, 70­72 funding Indonesia, 35, 154, 157t10.1 IFAD. see International Fund for Agricultural Development information, 136­137, 160 (IFAD) active and passive approaches to using, 146, 147b9.7 impact evaluations, 14, 125, 169 credibility of, 153, 170 impacts, 226 free flow of between levels, 48 impartiality of evaluations, 126­27, 169 internal and external use of, 19 implementation-based monitoring and evaluation systems, reaction to negative nature of, 46­47 98, 99­100, 99f6.3 strategies for sharing of, 146­50, 169­70 developing countries, 33 see also baselines; performance information key features of, 15­17 initiatives relationship to results monitoring, 101, 103, 103f6.5, internal, 10­11 104f6.6 for poverty reduction, 8­9 incentives, 41­42 see also international initiatives to learning, 34, 145b9.5 inputs, 1 and management of monitoring systems, 108 definition, 226 for M&E systems, 49, 53, 175­76 and financial resource monitoring, 37bi.x and readiness assessment, 165 links to outputs, 36bi.ix in sustaining M&E systems, 154, 155, 158b10.3, 170 measure of, 22 for use of findings, 146b9.6 and targets, 92­93, 94, 96 independent evaluation INSTAT. see Albanian Institute of Statistics (INSTAT) definition of, 226 institutional capacity, 21­22, 32 India, Andhra Pradesh Performance Accountability Act institutional development impact 244 Index definition of, 226 line graph, 137b8.2 institutional memory, 144, 145b9.5 line of sight, 48, 108, 139, 158 internal applications for monitoring and evaluation systems, logical framework 19­20 definition of, 226­27 internal demands, and readiness assessments, 44 internal evaluations, 22, 226, 31bi.viii M internal initiatives, public sector management, 10­11 macroeconomic indicators, 73 internal pressures, and evaluation issues, 27­28 Madagascar, 6 International Development Association (IDA) funding, 3, maintenance of monitoring systems, 107­8, 168 6­7 Malaysia, outcome-based budgeting, 35, 36bi.ix international development goals, 15 Mali, 34 International Fund for Agricultural Development (IFAD), management checklists, 155, 158b10.3, 158b10.4 use of evaluation information, 116­18 international initiatives, 3­8, 9 use of Gant chart in, 97f6.2, 97 International Monetary Fund (IMF), 73, 89b4.2 management information system, Brazil, 102b6.2 International Program in Development Evaluation Training management of monitoring systems, 107, 108, 168 (IPDET), 114­15 management tools, 83, 130­31, 132, 139 internet sites, to publish findings, 148 feedback, 130­31, 132, 139 interventions, 114 performance information as, 83 consensus for, 116 managers, and use of findings, 139 and evaluation information, 115­16, 128, 168 maps, 134 and impact evaluations, 125 maternal health and MDGs, 201 motivation for, 124 MBS. see Modified Budgeting System (MBS), Malaysia and outcome indicators, 65 MDBs. see Multilateral Development Banks (MDBs) IPDET. see International Program in Development Evalua- MDGs. see Millennium Development Goals (MDGs) tion Training (IPDET) measurements, frequency vs. precision of, 111, 112, 169 Ireland, 26, 28 media, empowerment of, 147­48 Italy, 28 meta-evaluation, 121, 125­26, 169, 227 Mexico, results-based monitoring, 100, 101b6.1 J midcourse corrections, 75 joint evaluations, 25, 150, 226 mid-term evaluation, 227 Jordan, e-government, 148 Millennium Development Goals (MDGs), 3­5, 72, 73 Jospin, Lionel, 30bi.vii adoption of, 25 list of, 200­203 K M&E systems integrated into, 54 knowledge, 163, 169 progress of, 92­93 findings promotion of, 140, 143­44, 146b9.6 ministries of finance incentives for, 146b9.6 Albania, 78b3.4 knowledge capital, 20 Egypt, 51b1.2 Korea, 27, 28, 31bi.viii Romania, 52b1.3 Kyrgyz Republic, 9, 33, 35, 49, 70 Uganda, 37bi.x Ministry of Planning, Budget, and Management, Brazil, L 102b6.2 laws, 52b1.3, 148 mixed approach to creating monitoring and evaluation and freedom of information, 148 systems, 26 Romania, 52b1.3 models leadership, 53 10-Step Model for Results-Based M&E System, 25fi.ii learning, 169 CREAM criteria, 68­70, 71f3.3, 166 findings promotion of, 140, 143­44, 146b9.6 enclave approach, 2, 24­25, 27, 35, 162, 163 incentives for, 146b9.6 for national development goals, 16, 18fi.i obstacles to, 144, 145b9.5 mixed approach, 2, 24­25, 163 Lebanon, and IMF data system, 89b4.2 whole-of-government, 24­25, 28, 29bi.vi, 35 lessons learned, definition, 226 Modified Budgeting System (MBS), Malaysia, 36bi.ix Index 245 monitoring, 24, 25fi.1, 39­40, 168 creating of evaluation cultures in, 163­64 Bangladesh, 50b1.1 identification of obstacles to learning, 144, 145b9.5 as complement to evaluation, 13­14 indications of progress in, 28­29 definition, 12, 227 M&E experience in, 27­28 examples of, 100f6.4 use of evaluations in, 15 and issues in choosing outcomes, 57­58 organizational culture, 145b9.5, 160b10.5, 160­61 key principles of building a system of, 103­5 outcome data, collection by government agencies, 156b10.2 levels of, 13 outcomes, 163, 166 overview, 96­98 and activities, 98 results-based, 99f6.3 conflicting evidence of, 120­21, 169 roles of, 13­15 definition, 227 types and levels of, 98­101, 101b6.1, 102b6.2 development of, 57­58, 59­60, 60f2.2, 61­64, 64f2.5 see also results-based monitoring system disaggregation of, 59­60, 67 Multilateral Development Banks (MDBs), 8 impact of design and implementation on, 119­20, 169 and implementation monitoring, 98, 99f6.3, 99­100 N importance of, 56­57 National Audit Office for the Parliament, United and indicators, 79b3.6 Kingdom, 149 link to work plans, 101, 103, 103f6.5, 104f6.6 National Council of Women, Egypt, 26, 51b1.2, 181 and targets, 95f5.3, 132, 133t8.1 national development goals, model for, 16, 18fi.i U.S. Department of Labor, 142b9.3 National Development Plans, 9, 35, 101b6.1 vs. outputs, 28 National Evaluation Policy, Sri Lanka, 77b3.3 see also indicators national goals, 48, 58, 61, 153, 165 outcome statements, 62f2.3, 63f2.4 national indicators, 70 outputs, 1 National Planning Authority, Uganda, 37bi.x achievement of, 16 National Poverty Reduction Strategies, 8, 46, 168 alignment with results, 99­100 Bangladesh, 50b1.1 definition, 227 and demand for M&E systems, 152 links to inputs, 36bi.ix and information sharing, 150 measure of, 22 National Poverty Reduction Strategy Papers (PRSPs), relationship to outcomes, 28, 57 7, 70 oversight, management, 102b6.2 National Strategy for Social and Economic Development oversight, parliamentary, 149 (NSSED), Albania, 78b3.4 ownership nation building, Malaysia, 36bi.ix of findings, 127 nongovernmental organizations (NGOs), xi, 1, 9, 10, 31, of M&E systems, 32, 45­46, 51b1.2, 53, 106­7, 168 39, 42, 48, 49, 50, 51, 53, 59, 77, 84, 88, 106, 147, 153, 154, 162, 174, 175, 176, 177, 188, 194, 210 P needs assessment, 41 participatory evaluations, 77b3.3, 227 negative information, 46­47 participatory process, in choosing outcomes, 58 the Netherlands, 27 partners and partnerships, 164, 168, 227 New York City, use of performance data to track crime, achieving results through, 105­6, 106f6.7 141b9.2 with civil servants, 139 NSSED. see National Strategy for Social and Economic and evaluation culture, 160b10.5 Development (NSSED), Albania formation of, 112 for global development, 202 O and incentives, 158b10.3 OECD. see Organisation for European Co-operation and inhibition of, 145b9.5 Development (OECD) intra-institutional, 105 oral presentations, 134 and sharing of information, 150 oral rehydration therapy (ORT), 16, 18fi.i Sri Lanka, 77b3.3 Organisation for European Co-operation and Development PEAP. see Poverty Eradication Action Plan (PEAP), (OECD), 2 Uganda conclusions and lessons from, 29, 32 perception, measure of, 69 246 Index performance Bangladesh, 50b1.1 definition, 227 PRSPs, 8­9 divergence between planned and actual, 118­19, 169 Uganda, 35, 37bi.x performance (continued) PPBS. see Program Performance Budgeting System (PPBS), linked to public expenditure framework, 34­35 Malaysia power in measuring of, 11­12, 163 predesigned indicators, 72­74, 166 Performance-Based Allocation system (IDA), 7 pre-implementation assessment, 121f7.4, 122, 169 performance framework/matrix, 64f2.5, 67f3.2, 81f4.2, 94, pretesting of data, 112, 168 95f5.3, 168 primary data, 83, 167 performance goals, 93, 160, 142b9.3, 156b10.2 privatization, 10, 162 performance indicators, 14ti.ii, 230n5 process evaluations and budget process, 30bi.vii definition of, 228 CREAM of, 68­70, 71f3.3, 166 process implementation assessment, 121f7.4, 122­23, 169 definition, 227 program evaluations, 13­14, 128, 128f7.6, 139b9.1, 143 identification of, 26 definition, 228 Romania, 52b1.3 and results-based M&E systems, 17, 19 setting of, 24, 166 program goals, 48, 58, 156b10.2 Sri Lanka, 77b3.3 program interventions, 13 use of, 75 program monitoring, examples of, 100f6.4 see also indicators program objectives, 228, 156b10.2 performance indicator targets, 91­93 Program Performance Budgeting System (PPBS), Malaysia, performance information, 47 36bi.ix in budget documents, 28, 29bi.vi progress, as qualitative indicator, 69­70 as management tool, 83 project evaluations, 13­14, 128, 128f7.6 and program evaluation, 13 definition, 228 sharing of, 104­5, 168 Korea, 31bi.viii source of demand for, 53 and results-based M&E systems, 17, 19 performance logic chain assessment, 121f7.4, 122, 169 project goals, 48, 58, 79b3.5, 158b10.4 performance measurement systems, 77, 141b9.2, 154, project monitoring, examples of, 100f6.4 156b10.2, 160 project objectives performance monitoring, 78b3.4, 227 definition of, 228 personnel, motivate, 139b9.1, 146b9.6 proxy indicators, 70­72, 166 pie chart, 137f8.2 PRSPs. see Poverty Reduction Strategy Papers (PRSPs) pilots public administration reforms, Malaysia, 36bi.ix Albania program, 26 public management, 11­12, 93, 170 and data collection, 87, 112 public officials, corruption among, 6bi.iii Egypt, 26, 51b1.2 public policies, 31bi.viii, 32 importance of conducting, 86­89 public sector, 44, 46, 52b1.3, 116, 169 of indicators, 167 public sector management Romania, 52b1.3 documenting progress of, 69­70 policy evaluations, 13, 17, 19, 128, 128f7.6 initiatives and forces for change in, 3­8, 10­11 policymakers, 21, 134­36 Korea, 31bi.viii policy monitoring, examples of, 100f6.4 and National Poverty Reduction Strategy, 8­9 policy planning, and developing countries, 32 overview, 2­3 politics public service, United Kingdom, 155b10.1 and impact of negative data, 46­47, 108 purpose, definition, 228 and M&E systems, 20­21, 33, 45 and setting of targets, 92, 93 Q polling data, 58 qualitative indicators, 69 Poverty Eradication Action Plan (PEAP), Uganda, 37bi.x quality assurance, 168, 228 poverty mapping, 88b4.1 quantitative indicators, 69 poverty reduction, 5, 200 Index 247 R technical challenges to, 21­22 rapid appraisal, 121f7.4, 123­26, 169 U.S. Department of Labor, 142b9.3 RBM. see results-based management (RBM) results findings readiness assessment, 23, 25fi.1, 39­40, 165 ten uses of, 139b9.1 readiness assessment survey, Annex I, 174­177 results information Bangladesh, 49, 50b1.1 active and passive approaches, 147b9.7 Egypt, 51b1.2, 53, 178­199 (Annex II) review government performance, 53­54 definition of, 228 key areas of, 43­48, 230n4 risk analysis Krygyz Republic, 35 definition of, 228 lessons learned in developing countries, 49­55 roles overview, 40­41, 230n3, 48­49 for assessing performance of government, 176­77 parts of, 41­43 and readiness assessment, 42 Romania, 52b1.3, 53 for sustaining M&E systems, 152­53, 170 reforms Romania, 52b1.3, 53, 54, 148 France, 30bi.vii rural areas, indicators for well-being of, 73 Malaysia, 36bi.ix Rural Development Indicators Handbook, World Bank, public sector, 46, 116, 169 73 reliability of data, 108, 109, 109f6.9, 109f6.10, 116, 168 Rural Score Card, 73 resource allocation, 28 Brazil, 102b6.2 S and evaluation information, 115, 120, 168, 169 secondary data, 83­84, 86, 167 Mexico, 101b6.1 sector goals, 48, 58, 61 and performance monitoring, 100 sector program evaluations and readiness assessment, 46 definition of, 228 resources self-evaluations, 31bi.viii, 229 level of, 92­93 service delivery, 16­17, 37bi.x management of, 96, 108 social indicators, 5, 149b9.8 and partnership formations, 105 sources, of data, 83­84 responsibilities Spain, 28 for assessing performance of government, 176­77 Sri Lanka, National Evaluation Policy 77b3.3, 204­210 and readiness assessment, 42, 165 (Annex IV) for sustaining M&E systems, 152­53, 170 staff performance appraisals, 158b10.3, 158b10.4 results-based management (RBM), 52b1.3, 128, 228 stakeholders, 1, 59, 124, 166 results-based monitoring and evaluation systems, 20, and accountability, 12, 160 99f6.3 consultation of, 23 capacity for, 21­22, 174­77 definition, 229 creation of, 46, 165­70 and demand for M&E systems, 32 as an emerging phenomenon, 162­64, 170 external and internal, 20 incentives and disincentives in sustaining of, 154, 155, and findings, 132 158b10.3, 158b10.4 identification of, 59 internal and external applications of, 19­20 involvement in evaluations, 126, 127, 169 key features of, 15­17, 103­5 monitoring performance of, 9 Mexico, 101b6.1 and number of indicators, 88 needs of, 106­8 and outcomes, 2­3, 58, 59, 67, 69 political challenges to, 20­21 and ownership of data, 106­7 project, program, and policy applications of, 17, 19 sharing information with, 146­50, 170 relationship to implementation monitoring, 101, 103, statistical capacity, 22 103f6.5, 104f6.6 strategic goals, 92, 142b9.3, 152, 154, 166 and stimulating cultural changes with, 160­61, strategic planning, developing countries, 33 160b10.5 Sub-Saharan African countries, 33 sustaining of, 152­54, 155, 155b10.1, 156b10.2, summative evaluation 157t10.1, 159, 170 definition of, 229 248 Index surveys, Albania, 88b4.1 Transparency International (TI), 3, 5, 6bi.iii, 50b1.1 sustainability, 12, 15, 229 Triangulation see also results-based monitoring and evaluation sys- definition of, 229 tems, sustaining of Tufte, Edward, 137f8.2 definition of, 229 tunnel vision, 144, 145b9.5 Turkey, 35 T tables, 135­36 U Tanzania, 6 Uganda, 6, 35, 37bi.x target groups, 60, 84, 229 United Kingdom, Citizen's Charters in, 154, 155b10.1 targets, 24, 35, 57 United Nations Development Programme (UNDP), 72, 83, Brazil's report on, 35 126 definition, 90­91 United Nations Educational, Scientific and Cultural Organ- formula for devising, 91f5.2 ization (UNESCO), 83 link to expenditures, 28 usefulness of evaluations, 126, 127, 169 link to work plans, 101, 103, 103f6.5, 104f6.6 U.S. General Accounting Office (GAO), 149 MDGs, 5bi.ii, 200­203 and outcomes, 132, 133t8.1 V performance framework/matrix for, 64f2.5, 67f3.2, validity 81f4.2, 94, 95f5.3, 168 of data, 109, 109f6.9, 109f6.10, 116, 168 for policy area, 95f5.3 definition, 229 related to development issues, 93­94, 94b5.1 of development hypotheses, 150 relationship to indicators, 3, 5bi.ii value for money (quality evaluations), 126­127 relationship to means and strategies, 99 vertical sharing of information, 104, 105, 168 selection of, 91­93, 167 viability, of monitoring and evaluation systems, 45, 170 technical adequacy of evaluations, 126­127, 169 visual presentations, 134­36, 137f8.2 technical assistance, 33, 166 technical capacity, 21­22, 33­34, 230n2 W technical training, Bangladesh, 50b1.1 web sites, to publish findings, 148 terms of reference welfare indicators, 76b3.2 definition of, 229 whole-of-government M&E model, 24­25, 28, 29bi.vi, 35 thematic evaluation women, 26, 51b1.2, 200 definition of, 229 workforce, 142b9.3 Three-Year Action Plan, Albania, 78b3.4 work plans, 97, 98 TI. see Transparency International (TI) outcomes and targets link to, 101, 103, 103f6.5, timeliness of data, 108, 109, 109f6.9, 109f6.10, 116, 168 104f6.6 training, 33, 50b1.1, 114­15 World Bank, 32, 73 transparency, 48, 147b9.7, 163 World Development Indicators, 73 culture of, 34 World Trade Organization (WTO), 3, 7 demand for, 44 written summaries, 133­134 demonstration of, 37, 140 WTO. see World Trade Organization (WTO) and HIPC, 9 provision of, 21, 24 Z and reforms, 10 Zambia, 33 and results-based M&E systems, 20 Ten Steps to a Results-Based Monitoring and Evaluation System Selecting Key Planning for Conducting a Indicators to Improvement -- Readiness Monitor Selecting Results The Role of Using Assessment Outcomes Targets Evaluations Findings 1 2 3 4 5 6 7 8 9 10 Agreeing on Baseline Data on Monitoring Reporting Sustaining the Outcomes to Indicators -- for Results Findings M&E System Monitor and Where Are We Within the Evaluate Today? Organization An effective state is essential to achieving socio-economic and sustain- able development. With the advent of globalization, there are growing pressures on governments and organizations around the world to be more responsive to the demands of internal and external stakeholders for good governance, accountability and transparency, greater development effectiveness, and delivery of tangible results. Govern- ments, parliaments, citizens, the private sector, nongovernmental organizations, civil society, international organizations, and donors are among the stakeholders interested in better performance. As demands for greater accountability and real results have increased, there is an attendant need for enhanced results-based monitoring and evaluation (M&E) of policies, programs, and projects. The focus of this handbook is on a comprehensive ten-step model that will help guide development practitioners through the process of designing and building a results-based M&E system. These steps begin with a "Readiness Assessment" and take the practitioner through the design, management, and, importantly, the sustainability of such sys- tems. The handbook describes each step in detail, the tasks needed to complete each, and the tools available to help along the way. THE WORLD BANK Africa Region Knowledge and Learning and Operations Evaluation Department 0-8213-5823-5