Publication: The Design and Implementation of a Menu of Evaluations
Loading...
Date
2011-01
ISSN
Published
2011-01
Author(s)
Editor(s)
Abstract
Policy makers and program managers are faced every day with major decisions resulting from insufficient funding, ongoing complaints about service delivery, unmet needs among different population groups, and limited results on the ground. There is a menu of evaluation types implemented by developing and Organization for Economic Co-operation Development (OECD) countries to tackle a wide range of policy and program management issues, considering time, resources and capacity constraints. International experience highlights the importance of a gradual approach when introducing evaluation tools into country-level monitoring and evaluation (M and E) systems. Different paths may work better for different countries depending on the main purpose of their M and E system, existing institutional capacity, the availability of funds, and external technical assistance.
Link to Data Set
Citation
“Rubio, Gloria M.. 2011. The Design and Implementation of a Menu of Evaluations. PREM Notes and Special Series on the Nuts and Bolts of Government M&E Systems; No. 6. © World Bank. http://hdl.handle.net/10986/11068 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Handbook on Impact Evaluation : Quantitative Methods and Practices(World Bank, 2010)This book reviews quantitative methods and models of impact evaluation. The formal literature on impact evaluation methods and practices is large, with a few useful overviews. Yet there is a need to put the theory into practice in a hands-on fashion for practitioners. This book also details challenges and goals in other realms of evaluation, including monitoring and evaluation (M&E), operational evaluation, and mixed-methods approaches combining quantitative and qualitative analyses. This book is organized as follows. Chapter two reviews the basic issues pertaining to an evaluation of an intervention to reach certain targets and goals. It distinguishes impact evaluation from related concepts such as M&E, operational evaluation, qualitative versus quantitative evaluation, and ex-ante versus ex post impact evaluation. Chapter three focuses on the experimental design of an impact evaluation, discussing its strengths and shortcomings. Various non-experimental methods exist as well, each of which are discussed in turn through chapters four to seven. Chapter four examines matching methods, including the propensity score matching technique. Chapter five deal with double-difference methods in the context of panel data, which relax some of the assumptions on the potential sources of selection bias. Chapter six reviews the instrumental variable method, which further relaxes assumptions on self-selection. Chapter seven examines regression discontinuity and pipeline methods, which exploit the design of the program itself as potential sources of identification of program impacts. Specifically, chapter eight presents a discussion of how distributional impacts of programs can be measured, including new techniques related to quantile regression. Chapter nine discusses structural approaches to program evaluation, including economic models that can lay the groundwork for estimating direct and indirect effects of a program. Finally, chapter ten discusses the strengths and weaknesses of experimental and non-experimental methods and also highlights the usefulness of impact evaluation tools in policy making.Publication Evaluating the Impact of Development Projects on Poverty : A Handbook for Practitioners(Washington, DC: World Bank, 2000-05)Very little is known about the actual impact of projects on the poor. Many are reluctant to carry out impact evaluations because they are deemed expensive, time consuming, and technically complex, and because the findings can be politically sensitive. Yet a rigorous evaluation can be powerful in assessing the appropriateness and effectiveness of programs. Evaluating impact is particularly critical in developing where resources are scarce and every dollar spent should aim to maximize its impact on poverty reduction. This handbook seeks to provide project managers and policy analysts with the tools needed for evaluating project impact. It is aimed at readers with a general knowledge of statistics. Chapter 1 presents an overview of concepts and methods, Chapter 2 discusses key steps and related issues to consider in implementation, Chapter 3 illustrates various analytical techniques through a case study, and Chapter 4 includes a discussion of lessons that have been reviewed for this handbook. The case studies, included in Annex I, were selected from a range of evaluations carried out by the Bank, other donor agencies, research institutions, and private consulting firms. Also included in the annexes are samples of the main components that would be necessary in planning any impact evaluation - sample terms of reference, a budget, impact indicators, a log frame, and a matrix of analysis.Publication Planning, Monitoring, and Evaluation : Methods and Tools for Poverty and Inequality Reduction Programs(World Bank, Washington, DC, 2013-01)As the author enter the second decade of the twenty-first century, governments, international organizations, nongovernmental organizations (NGOs), philanthropic organizations, and civil society groups worldwide are actively focusing on evidence-based policy and increased accountability to stakeholders (results agenda).The widespread implementation of the Results Agenda has generated a plethora of books, guides, academic papers, trainings, and case studies, which has enabled an ongoing maturation process in the field. Consequently, specialists are now better equipped to understand what works under which circumstances. Broadly speaking there are two interrelated questions which must be answered when assessing the sustainability of a government results agenda. First, is the institutional design and practice of government conducive to evidence-based policy making? Second, are the overarching monitoring and evaluation (M&E) methods and specific tools used appropriate for garnering the evidence demanded by government? These series of notes aim to make a small contribution to the latter question by summarizing and highlighting a selection of PM&E methods and the tools that governments and international organizations around the world have developed to put these into practice in their own contexts. The central goal of this initiative is to prompt a process of learning, reflection and action by providing practical information to those whose leadership role requires them to understand PM&E methods and their potential for enhancing evidence-based policy making.Publication Using Mixed Methods in Monitoring and Evaluation : Experiences from International Development(2010-03-01)This paper provides an overview of the various ways in which mixing qualitative and quantitative methods could add value to monitoring and evaluating development projects. In particular it examines how qualitative methods could address some of the limitations of randomized trials and other quantitative impact evaluation methods; it also explores the importance of examining "process" in addition to "impact", distinguishing design from implementation failures, and the value of mixed methods in the real-time monitoring of projects. It concludes by suggesting topics for future research -- including the use of mixed methods in constructing counterfactuals, and in conducting reasonable evaluations within severe time and budget constraints.Publication The Road to Results : Designing and Conducting Effective Development Evaluations(World Bank, 2009-12-01)The analytical, conceptual, and political framework of development is changing dramatically. The new development agenda calls for broader understandings of sectors, countries, development strategies, and policies. It emphasizes learning and continuous feedback at all phases of the development cycle. As the development agenda grows in scope and complexity, development evaluation follows suit. Development evaluator are moving away from traditional implementation and output-focused evaluation models toward results-based evaluation models, as the development community calls for results and embraces the millennium development goals. As the development community shifts its focus away from projects in order to comprehensively address country challenges, development evaluators are seeking methods with which to assess results at the country, sector, theme, policy, and even global levels. As the development community recognizes the importance of not only a comprehensive but also a coordinated approach to developing country challenges and emphasizes partnerships, development evaluators are increasingly engaged in joint evaluations. These joint evaluations, while advantageous in many respects, add to the complexity of development evaluation (OECD 2006). Additionally, development evaluators increasingly face the measurement challenge of determining the performance of an individual development organization in this broader context and of identifying its contribution. This text is intended as a tool for use in building development evaluation capacity. It aims to help development evaluators think about and explore the new evaluation architecture and especially to design and conduct evaluations that focus on results in meeting the challenges of development.
Users also downloaded
Showing related downloaded files
Publication The Mexican Social Protection System in Health(World Bank, Washington DC, 2013-01)With a population of 113 million and a per-capita Gross Domestic Product, or GDP of US$10,064 (current U.S. dollars), Mexico is one of the largest and highest-income countries in Latin America and the Caribbean (LAC). The country has benefited from sustained economic growth during the last decade, which was temporarily interrupted by the financial and economic crisis. Real GDP is projected to grow 3.8 percent and 3.6 percent in 2012 and 2013, respectively (International Monetary Fund, or IMF 2012). Despite this growth, poverty in the country remains high; with half of the population living below the national poverty line. The country is also highly heterogeneous, with large socioeconomic differences across states and across urban and rural areas. In 2010, while the extreme poverty ratio in the Federal District and the states of Colima and Nuevo Leon was below 3 percent, in Chiapas, Guerrero, and Oaxaca it was 25 percent or higher. These large regional differences are also found in other indicators of well-being, such as years of schooling, housing conditions, and access to social services. This case study assesses key features and achievements of the Social Protection System in Health (Sistema de Proteccion Social en Salud) in Mexico, and particularly of its main pillar, Popular Health Insurance (Seguro Popular, PHI). It analyzes the contribution of this policy to the establishment and implementation of universal health coverage in Mexico. In 2003, with the reform of the General Health Law, the PHI was institutionalized as a subsidized health insurance scheme open to the population not covered by the social security schemes. Today, the PHI covers all of its intended affiliates, about 52 million peoplePublication Guide to the Debt Management Performance Assessment Tool(Washington, DC, 2008-02-05)The purpose of this document is to provide guidance and supplemental information to assist with country assessments of debt management performance, using the Debt Management Performance Assessment (DeMPA) tool. The DeMPA is a methodology used for assessing public debt management performance through a comprehensive set of 15 performance indicators spanning the full range of government Debt Management (DeM) functions. It is based on the principles set out in the International Monetary Fund (IMF) and World Bank guidelines for public debt management, initially published in 2001 and updated in 2003. It is modeled after the Public Expenditure and Financial Accountability (PEFA) framework for performance measurement of public financial management. The DeMPA has been designed to be a user-friendly tool to undertake an assessment of the strengths and weaknesses in government DeM practices. This guide provides additional background and supporting information so that a no specialist in the area of debt management may undertake a country assessment effectively. The guide can be used by assessors in preparing for and undertaking an assessment. It is particularly useful for understanding the rationale for the inclusion of the indicators, the scoring methodology, and the list of supporting documents or evidence required, and the questions that could be asked for the assessment.Publication World Development Report 2008(Washington, DC, 2007)The world's demand for food is expected to double within the next 50 years, while the natural resources that sustain agriculture will become increasingly scarce, degraded, and vulnerable to the effects of climate change. In many poor countries, agriculture accounts for at least 40 percent of GDP and 80 percent of employment. At the same time, about 70 percent of the world's poor live in rural areas and most depend on agriculture for their livelihoods. World Development Report 2008 seeks to assess where, when, and how agriculture can be an effective instrument for economic development, especially development that favors the poor. It examines several broad questions: How has agriculture changed in developing countries in the past 20 years? What are the important new challenges and opportunities for agriculture? Which new sources of agricultural growth can be captured cost effectively in particular in poor countries with large agricultural sectors as in Africa? How can agricultural growth be made more effective for poverty reduction? How can governments facilitate the transition of large populations out of agriculture, without simply transferring the burden of rural poverty to urban areas? How can the natural resource endowment for agriculture be protected? How can agriculture's negative environmental effects be contained? This year's report marks the 30th year the World Bank has been publishing the World Development Report.Publication Sourcebook on the Foundations of Social Protection Delivery Systems(Washington, DC: World Bank, 2020-07-30)The Sourcebook synthesizes real-world experiences and lessons learned of social protection delivery systems from around the world, with a particular focus on social and labor benefits and services. It takes a practical approach, seeking to address concrete “how-to” questions, including: How do countries deliver social protection benefits and services? How do they do so effectively and efficiently? How do they ensure dynamic inclusion, especially for the most vulnerable and needy? How do they promote better coordination and integration—not only among social protection programs but also programs in other parts of government? How can they meet the needs of their intended populations and provide a better client experience? The Sourcebook structures itself around eight key principles that can frame the delivery systems mindset: (1) delivery systems evolve over time, do so in a non-linear fashion, and are affected by the starting point(s); (2) additional efforts should be made to “do simple well”, and to do so from the start rather than trying to remedy by after-the-fact adding-on of features or aspects; (3) quality implementation matters, and weaknesses in the design or structure of any core system element will negatively impact delivery; (4) defining the “first mile” for people interface greatly affects the system and overall delivery, and is most improved when that “first mile” is understood as the weakest link in delivery systems); (5) delivery systems do not operate in a vacuum and thus should not be developed in silos; (6) delivery systems can contribute more broadly to government’s ability to intervene in other sectors, such as health insurance subsidies, scholarships, social energy tariffs, housing benefits, and legal services; (7) there is no single blueprint for delivery systems, but there are commonalities and those common elements constitute the core of the delivery systems framework; (8) inclusion and coordination are pervasive and perennial dual challenges, and they contribute to the objectives of effectiveness and efficiency.Publication Classroom Assessment to Support Foundational Literacy(Washington, DC: World Bank, 2025-03-21)This document focuses primarily on how classroom assessment activities can measure students’ literacy skills as they progress along a learning trajectory towards reading fluently and with comprehension by the end of primary school grades. The document addresses considerations regarding the design and implementation of early grade reading classroom assessment, provides examples of assessment activities from a variety of countries and contexts, and discusses the importance of incorporating classroom assessment practices into teacher training and professional development opportunities for teachers. The structure of the document is as follows. The first section presents definitions and addresses basic questions on classroom assessment. Section 2 covers the intersection between assessment and early grade reading by discussing how learning assessment can measure early grade reading skills following the reading learning trajectory. Section 3 compares some of the most common early grade literacy assessment tools with respect to the early grade reading skills and developmental phases. Section 4 of the document addresses teacher training considerations in developing, scoring, and using early grade reading assessment. Additional issues in assessing reading skills in the classroom and using assessment results to improve teaching and learning are reviewed in section 5. Throughout the document, country cases are presented to demonstrate how assessment activities can be implemented in the classroom in different contexts.