64383 The World Bank JANUARY PREMnotes 2011 NUMBER 6 Special Series on The Design and Implementation of a Menu of Evaluations Gloria M. Rubio Policy makers and program managers are faced every day with major decisions resulting from insufficient funding, ongoing complaints about service delivery, unmet needs among different population groups, and limited results on the ground. There is a menu of evaluation types implemented by developing and Organization for Economic Co-operation Development (OECD) countries to tackle a wide range of policy and program management issues, considering time, resources and capacity constraints. International experience highlights the importance of a gradual approach when introducing evaluation tools into country-level M&E systems. Different paths may work better for different countries depending on the main purpose of their M&E system, existing institutional capacity, the availability of funds, and external technical assistance. Policy making and program management involve ation types to facilitate the selection of a mix of continuous decision making. Every day, policy evaluation tools depending on the information makers and program managers tackle questions and feedback needs for decision making at any such as how to allocate resources across different given moment in the lifespan of an intervention, interventions or program components, whether while also accounting for time, resource, and to add a new program or reform an existing one, capacity constraints. International experience il- how to choose between alternative service delivery lustrates uses of different evaluation tools across mechanisms, or how to improve program opera- countries and over time. tions. Any decision, even those favoring the status quo, entails some kind of assessment weighing Key Considerations in different courses of action. To what extent these Selecting an Evaluation Tool assessments are systematic and based on empiri- cal evidence influences the chances of making a Evaluations are valuable when properly conducted decision based on accurate information and being and focused on their intended purpose. Hence, able to take advantage of opportunities to improve the selection of the right evaluation tool is critical. policy interventions. Choosing an evaluation tool, or combination of Evaluation is the systematic collection and tools, depends on two main considerations. First, analysis of information to provide relevant feed- it depends on what information is needed to make back for decision making and policy formulation. major decisions. For example, is the evaluation There is a menu of evaluation types addressing a prompted by the need to have overall performance wide range of policy and program management information across programs to decide how to al- questions. This note discusses the policy relevance, locate resources? Or is it motivated by the need to application, and requirements of different evalu- have information on a particular program opera- FROM THE POVERTY REDUCTION AND ECONOMIC MANAGEMENT NETWORK tion to identify bottlenecks that hinder adequate 4. Who is the audience for the evaluation service delivery? Policy and program manage- information? Some evaluations may be de- ment issues and concerns drive the need for the manded internally within the organization evaluation, and in turn, point to the selection of by program management or policy makers a particular set of evaluation tools. in the ministry. Others may be externally The second consideration in selecting an required by the ministry of �nance, congress, evaluation tool is the availability of resources to or a donor organization. Internal and external conduct the evaluation. The menu of evaluation information needs may be different, leading tools includes different alternatives in terms of to different evaluation types. costs, duration, and capacity requirements. For 5. What resources (�nancial and human) are example, some evaluations tools provide rapid available to conduct the evaluation? Evalua- feedback at relatively lower costs, while others tion scope and tools should be aligned with require a longer implementation period, and available resources. In some cases, there may their costs vary accordingly. Although there are be an experienced team in charge of evalua- usually trade-offs between evaluation depth and tions with an earmarked budget. However, in rigor and resource requirements, the selection of most cases, resources devoted to evaluation an evaluation type allows some flexibility in ac- are limited or nonexistent, and institutional commodating particular circumstances. capacity is just developing. Key questions to ask before deciding on an evaluation tool, or combination of tools, include: 1. What is the purpose of the evaluation being Three Major Types of conducted, that is, what decision will be Program Evaluation informed by the evaluation? Once the key questions above have been an- 2. What kinds of information are needed to swered, an evaluation tool or tools can be se- make the decision? At what stage in the life lected. Evaluations can be classi�ed in several of the program will the evaluation be con- ways and some categories may overlap. For ex- ducted? These questions are closely related; ample, evaluations can be grouped according to information needs may vary depending the analytical paradigm (rationalist, pluralistic, on the program life stage. For example, a participative, and so forth), purpose (formative program that has just been planned would versus summative), content (goals, process, require an ex ante cost-bene�t analysis to outcomes/impact), time perspective (ex ante inform the decision on whether or not versus ex post), or the evaluator (internal versus to implement it. Alternatively, a newly external). Using a content-based classi�cation, implemented program would bene�t from this section discusses three major types of pro- information on how well operational pro- gram evaluation.1 Table 1 summarizes the policy cedures are followed, and whether there relevance, application, and requirements of each are any adjustments necessary for successful type of evaluation. program operation. 3. How quickly or (unexpectedly) is the in- Goal- or Objective-Based Evaluations formation needed? Sometimes evaluations Goal-based evaluations assess the clarity of a are planned well in advance and results are program’s objectives and its progress in achieving expected in due course, giving maximum these objectives. Questions asked by this type of flexibility in evaluation tool choice. In other evaluation include: cases, the selection of an evaluation tool is 1. How were program objectives established? influenced by the time frame of information Was the process effective and are the resulting demands from a particular policy process, objectives clear and appropriate? such as budget preparation. In many cases, 2. Is there a sound theory of change behind the however, information needs arise suddenly, program design? triggered by events that demand a quick 3. Are program inputs, activities, and outputs response, such as a macroeconomic crisis or aligned with the objectives or outcomes (in- reform opportunity. ternal consistency)? 2 PREMNOTE JANUARY 2011 Table 1. Main Evaluation Types Goal-based evaluations Process-based evaluations Impact evaluations What are the main questions answered - by this type of evaluation? - - pants? - - or to assess innovative pilot tion - - and analysis are analysis of survey and admin- required? with qualitative data analysis What skills are needed? possibly simple quantitative and qualitative methods methods methods 1– 3 months 1–6 months - it take? - Generally low Low to medium What are some Evaluation type of - evaluation? Additional - breadth and data availability Source JANUARY 2011 PREMNOTE 3 4. Are there sound performance indicators to �ciency in their achievement, results sustain- assess the program’s progress in achieving its ability, and institutional development impact. objectives? They are generally conducted at the end of the 5. What is the status of the program’s progress project by reviewing project documentation, toward achieving the objectives? visiting the borrowing country, and interview- 6. Will the goals be achieved according to the ing staff and government officials.4 timelines speci�ed in the program implemen- tation or operations plan? If not, why? Process-Based Evaluations 7. Do personnel have adequate resources Process-based evaluations are aimed at under- (money, equipment, facilities, training, and standing how a program works: What are the so forth) to achieve the goals? actual steps and activities involved in delivering 8. Do priorities need to be changed to ensure a good or a service? How close are they to agreed focus on achieving the goals? operation? Is program operation efficient? There 9. Should any goals be added or removed? Why? are numerous questions that might be asked in a This type of evaluation may be conducted process-based evaluation, including: at any time during program implementation. In 1. Is the program being implemented according early stages it could provide useful feedback on to design? program design, such as early warnings on lack 2. Are operational procedures appropriate to of clear objectives or inconsistencies between ensure the timely delivery of quality products resources, activities, and objectives. It may also or services? be a useful tool for a mid-term progress review 3. What is the level of compliance with the or project completion assessment. This type of operations manual? evaluation is usually based on a desk review of 4. Are there adequate resources (money, equip- program or project documentation and may be ment, facilities, training, and so forth) to complemented by analyzing monitoring data (if ensure the timely delivery of quality products available). In some cases, this evaluation includes or services? interviews with program staff to better under- 5. Are there adequate systems (human re- stand the goal-setting process and the enabling sources, �nancial, management information, or hindering factors behind observed progress. and so forth) in place to support program These are generally low-cost evaluations that can operations? be completed in a short period of time. 6. Are program clients receiving quality prod- Examples of this type of evaluation include: ucts and services? • Mexico, Design Evaluations: This evaluation 7. What is the general process that program analyzes new programs’ design, including their clients go through with the product or pro- link to national development objectives, inter- gram? Are program clients satis�ed with the nal consistency, alignment between program processes and services? log frame and operation rules, and potential 8. Are there any operational bottlenecks? synergy or duplicity with other programs. It 9. Is the program reaching the intended popu- relies on a desk review of program documenta- lation? Are program reach-out activities tion, including the log frame.2 adequate to ensure the desired level of target • Mexico, Performance-Specific Evaluation— population participation? This evaluation assess social programs’ prog- This type of evaluation is useful in a variety ress in meeting their objectives and annual of circumstances. For example, process-based targets based on a desk review of the Perfor- evaluations are helpful to obtain early warnings mance Evaluation System (SED) data. These of operational difficulties in newly implemented evaluations are conducted annually for about programs or components, particularly among 120 different programs and cost on average those involving complex procedures. It may also US$5,500.3 be conducted at regular intervals to check that • World Bank, Independent Evaluation Group, operation remains on track and follows estab- Project Performance Assessment—These as- lished procedures, or at any time when there are sessments evaluate the relevance of projects’ staff or client complaints about service delivery. objectives, whether goals are being met, the ef- Furthermore, there may be fears of inefficiency 4 PREMNOTE JANUARY 2011 in long-existing programs that warrant a process- be complemented with qualitative analysis. based evaluation. Compared to other evaluation types, impact Process-based evaluation tools vary consider- evaluations require more time, technical skills, and ably in depth and breadth, and thus in their costs are costlier. Hence, they are most effective when and time required. They range from qualitative applied selectively to answer strategic policy ques- assessments in a few program sites to user or fa- tions or to assess innovative pilot interventions cilities surveys involving representative samples. testing a new, unproven, but promising approach. Process-based evaluations use a combination of Although the impact evaluation process can start data collection and analysis methods including: very early in a program life, results are not usually interviews with program staff and clients; user and available until after several months of program facility surveys; focus groups; direct observation; implementation. Different outcomes of interest record review; and analysis of monitoring data. require varying exposure and maturation times for measurement. For example, changes in household Impact Evaluations consumption can be measured after six months An impact evaluation assess if a program is pro- of program implementation, but changes in nutri- ducing the intended outcomes through provid- tion outcomes require a longer period. ing relevant, quality outputs and services to the Impact evaluation results combined with targeted population. Outcomes are the changes in program cost analysis create an additional evalu- well-being experienced by individuals as a result ation tool. Cost-effectiveness analysis compares of participating in the program. They are differ- the relative performance of two or more programs ent from program outputs, which are measured or program alternatives in achieving a common by the number of goods or services delivered or outcome. Once impact and cost information the number of people served. Outcomes involve are available across a variety of programs, cost- bene�cial transformations in participants’ knowl- effectiveness analysis will allow policy makers to edge, attitudes, values, skills, behaviors, condition, make informed decisions on which intervention or status. Intended outcomes vary depending to invest in. on program objectives and design, for example, Resources for impact evaluations include: increased literacy, improved nutrition, decreased • The World Bank Impact Evaluation Web site, disease incidence, and so on. The main questions which includes methodological and imple- asked in an impact evaluation are: mentation guidelines for conducting impact 1. Does the program or policy have the desired evaluations as well as a database of evaluations effects on individuals, households, or institu- of World Bank–supported interventions. tions? • The Poverty Action Lab at MIT, which pro- 2. Are these effects attributable to the program motes the use of randomized evaluations to or would they have occurred anyway? answer questions critical to poverty allevia- 3. Are there any unintended consequences, tion by providing methodological guidance, a positive or negative, on program participants? randomized evaluations database, and policy 4. Are program costs justi�ed in terms of its lessons from cost-effectiveness analysis. welfare impact? In addition to the three types of evaluation 5. When there are various program implemen- discussed here, there are other analytical tools and tation alternatives, which one is the most approaches that are widely used. Box 1 discusses cost-effective? briefly cost-bene�t analysis and box 2 provides a Impact evaluations are useful to inform a glance to participatory evaluation. range of policy decisions, from scaling up effec- tive interventions, to adjusting program design, to curtailing unpromising interventions. In addition, Selecting a Mix of Evaluation they help generate evidence on which approach is Tools: International Experience more effective in reaching a particular objective A number of monitoring and evaluation (M&E) when comparing different programs or different systems at the country level use a combination of intervention options within a program. evaluation tools. The combinations vary across Impact evaluations rely on quantitative countries and have evolved over time. Country methods (statistics and econometrics), but can experiences show that the selection of an evalu- JANUARY 2011 PREMNOTE 5 Box 1. Another Analytical Tool Box 2. A Participatory Approach to Evaluation - - vention bene�ts in monetary terms and detailed - Source ation tool responds to differences in the driving principles of their M&E systems: transparency and accountability; policy and spending effective- ness and efficiency; or improved program man- agement. The type of tool selected also reflects - myriad country conditions, such as institutional capacity, resource availability, and institutional arrangements. Most countries have followed an in- cremental approach in the use of evaluation tools. In Chile, for example, the Budget Department - (Dirección de Presupuesto, DIPRES) at the Min- istry of Finance is in charge of the Management Evaluation and Control System (MECS). Its main emphasis is to promote the efficient allocation of public spending, and it uses various M&E tools that were gradually added between 1994 and 2002. DI- Source PRES made a conscious decision to start simple by incorporating the monitoring of performance in- dicators into the budget process. In 1997, DIPRES ability coupled with the demonstration effect began conducting a relatively low-cost and rapid of PROGRESA’s (the Education, Health, and evaluation known as the Government Program Nutrition Program) impact evaluation.5 Start- Evaluation (box 3). Since then, these evaluations ing in 2000, the Appropriations Bill required have been consistently used to inform the budget an annual external evaluation for all federal process. In 2001, impact evaluations were added programs subject to rules of operation. At the to the Chilean M&E system. Programs are selected time, evaluation guidelines did not distinguish for an impact evaluation if they involve a substan- between different types of evaluations. However, tial amount of public resources and there is no every year external evaluations were expected to evidence from previous evaluations of their �nal answer questions ranging from compliance with outcome. DIPRES and the Congress jointly decide operation rules, to program impact, to program which programs should be evaluated each year. cost bene�ts. This was an overwhelming demand In Mexico, the M&E system stems from an given the nascent evaluation institutional capacity, increasing demand for transparency and account- both from the demand (ministries) and the supply side (universities and research institutions). Box 3. Overall Performance Evaluation Then, in 2007, the newly created National Evaluation Council (CONEVAL), the Min- istry of Finance (MoF), and the Ministry of Public Management (MoPM) jointly issued a revised set of guidelines including a set of evaluations tools and the criteria for tool application. In addition, the guidelines established the publication of an Annual Program Evaluation (PAE) that speci�es which programs are required to have a particular type of evaluation, depending on their strategic relevance, particular policy interest, and previous evaluation results. The selection of programs and evaluation tools is made jointly by CONEVAL, MoF, - and MoPM. Ministries can also propose additional evaluations to support policy or management decisions or to get speci�c - program performance information. Colombia’s M&E system, SINERGIA, was launched in 1994. Its main focus is on national planning and accountability objec- tives. The system’s original design included Government a monitoring scheme and a program evalu- ation component; however, the evaluation component did not become operational - until 2002. Since then, SINERGIA’s two main components consist of a system of performance indicators that tracks progress against the National Development Plan goals and an agenda of rigorous impact eval- uations. Technical assistance and funding from donors allowed Colombia to opt for a more complex and costlier evaluation tool. - Recently Colombia incorporated a lower cost and more rapid alternative known as Executive Evaluations (box 3). - Key Lessons - Using the appropriate tool, evaluations can help address a number of different manage- ment issues. The appropriate evaluation tool depends on the intended use of the evaluation findings, budget availability, time available, and capacity constraints. There are usually trade-offs between evaluation depth and rigor and resource requirements. Goal-based evaluations are quick and low cost, but tend to have data Source: limitations and less rigorous analysis. By JANUARY 2011 PREMNOTE 7 contrast, impact evaluations are more rigorous, as Research Associate at the Public Policy Institute but require considerable technical expertise, take in the Universidad de las Americas in Puebla. She longer to yield results, and cost more. Moreover, it has collaborated with the World Bank as a con- is important to realize that any particular type of sultant and staff member in Washington, DC. evaluation can be conducted at a higher or lower She holds a master’s degree in Public Affairs from analytical level, and thus costs vary accordingly. the Woodrow Wilson School at Princeton University. Another important lesson is that evaluations Currently, she works as a consultant, providing are not a once in a lifetime exercise, nor does the technical assistance in designing and implement- use of one evaluation type exclude the rest. Policy ing M&E systems and conducting analytical work and program information needs are continuous on social policy issues. and change over time, so can the evaluation tool. Moreover, evaluation tools are complements Notes rather than substitutes. When they are combined, 1. There are other types of evaluations focused on they provide an even deeper understanding of other levels of public sector, including organizations programs and policies strengths and weaknesses. and sectors that are not considered in this note. For example, whereas impact evaluations can 2. See Web site for more information on Mexico’s produce reliable estimates of the causal effects of design evaluations, http://www.coneval.gob.mx/ a program, they are not typically designed to lend coneval2/htmls/evaluacion_monitoreo/HomeEval- insights into program implementation. Monitor- Monitoreo.jsp?categorias=EVAL_MON,EVAL_ ing data and process evaluations are needed to MON-diseno. track program implementation and examine ques- 3. See Web site for more information on Mexico’s tions of process that are critical to informing and performance-specific evaluation, http://www. interpreting the results from impact evaluations. coneval.gob.mx/. International experience highlights the impor- 4. See Web site for more information on the World tance of gradually incorporating evaluation tools Bank’s Project Performance Assessments, http:// into country-level M&E systems. Countries will web.worldbank.org/external/default/main?startP oint=0&startDate=2008%2F01%2F01&theSiteP require different evaluation tools depending on K=1324361&piPK=64254724&pagePK=642545 the main purpose of their M&E system, existing 14&menuPK=64253143. institutional capacity, and the availability of funds 5. PROGRESA, now called Oportunidades, is a and external technical assistance. In some cases, conditional cash transfer program launched in using less complex evaluation tools, but ensuring 1997 as an innovative pilot intervention to reduce they are properly applied and providing consistent poverty through human capital investment. The feedback, may be an appropriate strategy to build program impact evaluation became a role model for an M&E system. In other cases, a few properly other social development interventions in Mexico. conducted and highly relevant impact evalua- tions with large demonstration effects may be a Bibliography good way to motivate a shift toward results-based Departamento Nacional de Planeación. 2008. Evalu- management. ación Ejecutiva Lineamientos Metodológicos. Repúbli- Finally, as important as it is to select the right ca de Colombia mix of evaluation tools, it is also important that División de Control de Gestión. 2009. Metodología evaluation results are used to inform policy and Evaluación de Impacto. Gobierno de Chile, Ministe- program decision making. Decision making in- rio de Hacienda, Dirección de Presupuestos formed and supported by reliable and systematic Gertler, Paul J., Sebastian Martinez, Patrick Premand, evaluations is more likely to lead to the success of Laura B. Rawlings, and Christel M. J. Vermeersch. 2010. Impact Evaluation in Practice. World Bank, policy and program interventions. Washington, DC. Jackson, Edward T., and Yusuf Kassam, eds. 1998. About the Author Knowledge Shared: Participatory Evaluation in De- velopment Cooperation. Kumarian Press. Gloria M. Rubio is the former Director General McNamara, Carter. 2006. Field Guide to Nonprofit of the Social Programs Evaluation and Monitoring Program Design, Marketing and Evaluation, 4 th Unit at the Ministry of Social Development in Edition. Minneapolis, Minnesota: Authenticity Mexico. Prior to joining the ministry, she worked Consulting, LLC. 8 PREMNOTE JANUARY 2011 Zaltsman, Ariel. 2006. “Experience with Institution- Web Sites alizing Monitoring and Evaluation in Five Latin Colombia, SINERGIA: http://sinergia.dnp.gov.co/ American Countries: Argentina, Chile, Colombia, PortalDNP/ Costa Rica, and Uruguay.� ECD Working Paper Chile, DIPRES: http://www.dipres.cl/572/proper- Series 16, Independent Evaluation Group, World tyvalue-15697.html Bank, Washington, DC. Mexico, CONEVAL: http://www.coneval.gob.mx/ This note series is intended to summarize good practices and key policy �ndings on PREM-related topics. The views expressed in the notes are those of the authors and do not necessarily reflect those of the World Bank. PREMnotes are widely distributed to Bank staff and are also available on the PREM Web site (http://www. worldbank.org/prem). If you are interested in writing a PREMnote, email your idea to Madjiguene Seck at mseck@worldbank.org. For additional copies of this PREMnote please contact the PREM Advisory Service at x87736. This series is for both external and internal dissemination JANUARY 2011 PREMNOTE 9