Publication: Handbook on Impact Evaluation :
Quantitative Methods and Practices
Loading...
Published
2010
ISSN
Date
2012-03-19
Editor(s)
Abstract
This book reviews quantitative methods and models of impact evaluation. The formal literature on impact evaluation methods and practices is large, with a few useful overviews. Yet there is a need to put the theory into practice in a hands-on fashion for practitioners. This book also details challenges and goals in other realms of evaluation, including monitoring and evaluation (M&E), operational evaluation, and mixed-methods approaches combining quantitative and qualitative analyses. This book is organized as follows. Chapter two reviews the basic issues pertaining to an evaluation of an intervention to reach certain targets and goals. It distinguishes impact evaluation from related concepts such as M&E, operational evaluation, qualitative versus quantitative evaluation, and ex-ante versus ex post impact evaluation. Chapter three focuses on the experimental design of an impact evaluation, discussing its strengths and shortcomings. Various non-experimental methods exist as well, each of which are discussed in turn through chapters four to seven. Chapter four examines matching methods, including the propensity score matching technique. Chapter five deal with double-difference methods in the context of panel data, which relax some of the assumptions on the potential sources of selection bias. Chapter six reviews the instrumental variable method, which further relaxes assumptions on self-selection. Chapter seven examines regression discontinuity and pipeline methods, which exploit the design of the program itself as potential sources of identification of program impacts. Specifically, chapter eight presents a discussion of how distributional impacts of programs can be measured, including new techniques related to quantile regression. Chapter nine discusses structural approaches to program evaluation, including economic models that can lay the groundwork for estimating direct and indirect effects of a program. Finally, chapter ten discusses the strengths and weaknesses of experimental and non-experimental methods and also highlights the usefulness of impact evaluation tools in policy making.
Link to Data Set
Citation
“Khandker, Shahidur R.; Koolwal, Gayatri B.; Samad, Hussain A.. 2010. Handbook on Impact Evaluation :
Quantitative Methods and Practices. © World Bank. http://hdl.handle.net/10986/2693 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Evaluating the Impact of Development Projects on Poverty : A Handbook for Practitioners(Washington, DC: World Bank, 2000-05)Very little is known about the actual impact of projects on the poor. Many are reluctant to carry out impact evaluations because they are deemed expensive, time consuming, and technically complex, and because the findings can be politically sensitive. Yet a rigorous evaluation can be powerful in assessing the appropriateness and effectiveness of programs. Evaluating impact is particularly critical in developing where resources are scarce and every dollar spent should aim to maximize its impact on poverty reduction. This handbook seeks to provide project managers and policy analysts with the tools needed for evaluating project impact. It is aimed at readers with a general knowledge of statistics. Chapter 1 presents an overview of concepts and methods, Chapter 2 discusses key steps and related issues to consider in implementation, Chapter 3 illustrates various analytical techniques through a case study, and Chapter 4 includes a discussion of lessons that have been reviewed for this handbook. The case studies, included in Annex I, were selected from a range of evaluations carried out by the Bank, other donor agencies, research institutions, and private consulting firms. Also included in the annexes are samples of the main components that would be necessary in planning any impact evaluation - sample terms of reference, a budget, impact indicators, a log frame, and a matrix of analysis.Publication Using Mixed Methods in Monitoring and Evaluation : Experiences from International Development(2010-03-01)This paper provides an overview of the various ways in which mixing qualitative and quantitative methods could add value to monitoring and evaluating development projects. In particular it examines how qualitative methods could address some of the limitations of randomized trials and other quantitative impact evaluation methods; it also explores the importance of examining "process" in addition to "impact", distinguishing design from implementation failures, and the value of mixed methods in the real-time monitoring of projects. It concludes by suggesting topics for future research -- including the use of mixed methods in constructing counterfactuals, and in conducting reasonable evaluations within severe time and budget constraints.Publication Planning, Monitoring, and Evaluation : Methods and Tools for Poverty and Inequality Reduction Programs(World Bank, Washington, DC, 2013-01)As the author enter the second decade of the twenty-first century, governments, international organizations, nongovernmental organizations (NGOs), philanthropic organizations, and civil society groups worldwide are actively focusing on evidence-based policy and increased accountability to stakeholders (results agenda).The widespread implementation of the Results Agenda has generated a plethora of books, guides, academic papers, trainings, and case studies, which has enabled an ongoing maturation process in the field. Consequently, specialists are now better equipped to understand what works under which circumstances. Broadly speaking there are two interrelated questions which must be answered when assessing the sustainability of a government results agenda. First, is the institutional design and practice of government conducive to evidence-based policy making? Second, are the overarching monitoring and evaluation (M&E) methods and specific tools used appropriate for garnering the evidence demanded by government? These series of notes aim to make a small contribution to the latter question by summarizing and highlighting a selection of PM&E methods and the tools that governments and international organizations around the world have developed to put these into practice in their own contexts. The central goal of this initiative is to prompt a process of learning, reflection and action by providing practical information to those whose leadership role requires them to understand PM&E methods and their potential for enhancing evidence-based policy making.Publication Integrating Qualitative Methods into Investment Climate Impact Evaluations(World Bank Group, Washington, DC, 2014-12)Incorporating qualitative methods into the evaluation of development programs has become increasingly popular in recent years, both for the distinctive insights such approaches can bring in their own right and because of their capacity to complement the strengths -- and where necessary correct some of the weaknesses -- of quantitative approaches. Some initial work deploying mixed methods has been undertaken in the assessment of investment climate reforms, but considerable room for expansion exists. This paper summarizes some of the key principles and practices underpinning mixed methods evaluations in development, highlight some notable examples of how such work has been conducted (and the particular contributions it has made), and offers some guidelines for those seeking to increase the sophistication and utility of qualitative methods in the evaluation of investment climate reforms.Publication Impact Evaluation in Practice, First Edition(World Bank, 2011)The Impact Evaluation in Practice handbook is a comprehensive and accessible introduction to impact evaluation for policymakers and development practitioners. The book incorporates real-world examples to present practical guidelines for designing and implementing evaluations. Readers will gain an understanding of the uses of impact evaluation and the best ways to use evaluations to design policies and programs that are based on evidence of what works most effectively. The handbook is divided into three sections: Part One discusses what to evaluate and why; Part Two outlines the theoretical underpinnings of impact evaluation; and Part Three examines how to implement an evaluation. Case studies illustrate different methods for carrying out impact evaluations.
Users also downloaded
Showing related downloaded files
Publication Commodity Markets Outlook, October 2024(Washington, DC: World Bank, 2024-10-29)Commodity prices are expected to decrease by 5 percent in 2025 and 2 percent in 2026. The projected declines are led by oil prices but tempered by price increases for natural gas and a stable outlook for metals and agricultural raw materials. The possibility of escalating conflict in the Middle East represents a substantial near-term upside risk to energy prices, with potential knock-on consequences for other commodities. However, over the forecast horizon, longer-term dynamics—including decelerating global oil demand, diversifying oil production, and ample oil supply capacity—suggest sizable downside risks to oil prices, especially if OPEC+ unwinds its latest production cuts. There are also dual risks to industrial commodity demand stemming from economic activity. On the one hand, concerted stimulus in China and above-trend growth in the United States could push commodity prices higher. On the other, weaker-than-anticipated global industrial activity could dampen them. Following several overlapping global shocks in the early 2020s, which drove parallel swings in commodity prices, commodity markets appear to be departing from a period of tight synchronization. A Special Focus analyzes commodity price synchronization over time and considers the relative importance across commodity cycles of a wide range of demand and supply shocks, including global demand shocks and shocks specific to different commodity markets. It concludes that, while supply shocks were the dominant commodity price driver in the early 2000s and around the global financial crisis, post-pandemic price movements have been more substantially shaped by commodity-specific shocks, such as those related to conflicts.Publication Impact Evaluation in Practice, Second Edition(Washington, DC: Inter-American Development Bank and World Bank, 2016-09-13)The second edition of the Impact Evaluation in Practice handbook is a comprehensive and accessible introduction to impact evaluation for policy makers and development practitioners. First published in 2011, it has been used widely across the development and academic communities. The book incorporates real-world examples to present practical guidelines for designing and implementing impact evaluations. Readers will gain an understanding of impact evaluations and the best ways to use them to design evidence-based policies and programs. The updated version covers the newest techniques for evaluating programs and includes state-of-the-art implementation advice, as well as an expanded set of examples and case studies that draw on recent development challenges. It also includes new material on research ethics and partnerships to conduct impact evaluation. The handbook is divided into four sections: Part One discusses what to evaluate and why; Part Two presents the main impact evaluation methods; Part Three addresses how to manage impact evaluations; Part Four reviews impact evaluation sampling and data collection. Case studies illustrate different applications of impact evaluations. The book links to complementary instructional material available online, including an applied case as well as questions and answers. The updated second edition will be a valuable resource for the international development community, universities, and policy makers looking to build better evidence around what works in development.Publication Women’s Empowerment in Action(World Bank, Washington, DC, 2018-12)Women in developing countries are disempowered: high youth unemployment, early marriage and childbearing interact to limit their investments into human capital and enforce dependence on men. The authors evaluate a multi-faceted policy intervention attempting to jumpstart adolescent women’s empowerment in Uganda, a context in which 60 percent of the population are aged below twenty. The intervention aims to relax human capital constraints that adolescent girls face by simultaneously providing them vocational training and information on sex, reproduction and marriage. The authors find that four years post-intervention, adolescent girls in treated communities are 48 percent more likely to engage in income generating activities, an impact almost entirely driven by their greater engagement in self-employment. Teen pregnancy falls by 34 percent, and early entry into marriage/cohabitation falls by 62 percent. Strikingly, the share of girls reporting sex against their will drops by close to a third and aspired ages at which to marry and start childbearing move forward. The results highlight the potential of a multi-faceted program that provides skills transfers as a viable and cost-e¤ective policy intervention to improve the economic and social empowerment of adolescent girls over a four year horizon.Publication Global Economic Prospects, January 2024(Washington, DC: World Bank, 2024-01-09)Note: Chart 1.2.B has been updated on January 18, 2024. Chart 2.2.3 B has been updated on January 14, 2024. Global growth is expected to slow further this year, reflecting the lagged and ongoing effects of tight monetary policy to rein in inflation, restrictive credit conditions, and anemic global trade and investment. Downside risks include an escalation of the recent conflict in the Middle East, financial stress, persistent inflation, weaker-than-expected activity in China, trade fragmentation, and climate-related disasters. Against this backdrop, policy makers face enormous challenges. In emerging market and developing economies (EMDEs), commodity exporters face the enduring challenges posed by fiscal policy procyclicality and volatility, which highlight the need for robust fiscal frameworks. Across EMDEs, previous episodes of investment growth acceleration underscore the critical importance of macroeconomic and structural policies and an enabling institutional environment in bolstering investment and long-term growth. At the global level, cooperation needs to be strengthened to provide debt relief, facilitate trade integration, tackle climate change, and alleviate food insecurity.Publication Indigenous Latin America in the Twenty-First Century(2015)In 2013 the World Bank set itself two ambitious goals: to end extreme poverty within a generation and to boost the prosperity of the bottom 40 percent of the population worldwide. In Latin America, the significance of both goals cannot be overstated. Indigenous people account for about 8 percent of the population, but represent 14 percent of the poor and over 17 percent of all Latin Americans living on less than United States (U.S.) $2.50 a day. Though the World Bank has chosen two general indicators for measuring progress toward its twin goals - the proportion of people living on less than U.S. $1.25 a day (purchasing power parity, 2005) and the growth of real capital income among the bottom 40 percent of the population - this report acknowledges that these indicators offer only a partial view of the obstacles preventing many indigenous peoples from achieving their chosen paths of development. The report notes that in Bolivia, Quechua women are 28 percent less likely to complete secondary school than a nonindigenous Bolivian woman, while Quechua men are 14 percent less likely to complete secondary school than non-indigenous men. This report seeks to contribute to these discussions by offering a brief, preliminary glance at the state of indigenous peoples in Latin America at the end of the first decade of the millennium. The authors believe that this is the first, necessary step to start working on a concerted and evidence-based agenda for subsequent work in critical areas of development such as education, health, and land rights. The report makes a critical analysis of the many inconsistencies present in much of the data, which in many cases are intrinsic to the difficulties of approaching indigenous issues with tools and data sets not originally intended to account for or include indigenous peoples’ voices and special needs. The report is divided into six sections. The first part, how many and where they are provides a demographic overview of indigenous people in the region, including population, geographic distribution, number of ethnic groups, and indigenous languages. The second section, mobility, migration, and urbanization describes a growing tendency among indigenous people to migrate to Latin American cities, which are becoming critical, though largely ignored, areas for political participation, and market articulation. The third section, development with identity briefly discusses the concept of poverty and reflects on how the use of predominantly Western indicators of well-being might condition the understanding of indigenous peoples’ situations and needs. The fourth and fifth sections broaden this argument by focusing on two particular instances of exclusion - the market and education.