Publication: Combining Quantitative and Qualitative Methods for Program Monitoring and Evaluation: Why Are Mixed-Method Designs Best?
Loading...
Date
2011-05
ISSN
Published
2011-05
Author(s)
Editor(s)
Abstract
Despite significant methodological advances, much program evaluation and monitoring data are of limited utility because of an over-reliance on quantitative methods alone. While surveys provide generalizable findings on what outcomes or impacts have or have not occurred, qualitative methods are better able to identify the underlying explanations for these outcomes and impacts, and therefore enable more effective responses. Qualitative methods also inform survey design, identify social and institutional drivers and impacts that are hard to quantify, uncover unanticipated issues, and trace impact pathways. When used together, quantitative and qualitative approaches provide more coherent, reliable, and useful conclusions than do each on their own. This note identifies key elements of good mixed-method design and provides examples of these principles applied in several countries.
Link to Data Set
Citation
“Adato, Michelle. 2011. Combining Quantitative and Qualitative Methods for Program Monitoring and Evaluation: Why Are Mixed-Method Designs Best?. PREM Notes and Special Series on the Nuts and Bolts of Government M&E Systems; No. 9. © World Bank. http://hdl.handle.net/10986/11063 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Measurement and Meaning : Combining Quantitative and Qualitative Methods for the Analysis of Poverty and Social Exclusion in Latin America(Washington, DC: World Bank, 2001-12)This report consists of a collection of case studies from Latin America combining qualitative and quantitative research methods for the analysis of poverty within a social exclusion framework. The first chapter provides an overview of the differences between quantitative and qualitative methods, and the gains from using both types of methods in applied work. The other chapters are devoted to three case studies on reproductive health in rural Argentina, the targeting of social programs in Chile, and social exclusion in urban Uruguay. Each case study was prepared within the broader context of country-specific economic and sectoral work at the World Bank.Publication Handbook on Impact Evaluation : Quantitative Methods and Practices(World Bank, 2010)This book reviews quantitative methods and models of impact evaluation. The formal literature on impact evaluation methods and practices is large, with a few useful overviews. Yet there is a need to put the theory into practice in a hands-on fashion for practitioners. This book also details challenges and goals in other realms of evaluation, including monitoring and evaluation (M&E), operational evaluation, and mixed-methods approaches combining quantitative and qualitative analyses. This book is organized as follows. Chapter two reviews the basic issues pertaining to an evaluation of an intervention to reach certain targets and goals. It distinguishes impact evaluation from related concepts such as M&E, operational evaluation, qualitative versus quantitative evaluation, and ex-ante versus ex post impact evaluation. Chapter three focuses on the experimental design of an impact evaluation, discussing its strengths and shortcomings. Various non-experimental methods exist as well, each of which are discussed in turn through chapters four to seven. Chapter four examines matching methods, including the propensity score matching technique. Chapter five deal with double-difference methods in the context of panel data, which relax some of the assumptions on the potential sources of selection bias. Chapter six reviews the instrumental variable method, which further relaxes assumptions on self-selection. Chapter seven examines regression discontinuity and pipeline methods, which exploit the design of the program itself as potential sources of identification of program impacts. Specifically, chapter eight presents a discussion of how distributional impacts of programs can be measured, including new techniques related to quantile regression. Chapter nine discusses structural approaches to program evaluation, including economic models that can lay the groundwork for estimating direct and indirect effects of a program. Finally, chapter ten discusses the strengths and weaknesses of experimental and non-experimental methods and also highlights the usefulness of impact evaluation tools in policy making.Publication Integrating Qualitative Methods into Investment Climate Impact Evaluations(World Bank Group, Washington, DC, 2014-12)Incorporating qualitative methods into the evaluation of development programs has become increasingly popular in recent years, both for the distinctive insights such approaches can bring in their own right and because of their capacity to complement the strengths -- and where necessary correct some of the weaknesses -- of quantitative approaches. Some initial work deploying mixed methods has been undertaken in the assessment of investment climate reforms, but considerable room for expansion exists. This paper summarizes some of the key principles and practices underpinning mixed methods evaluations in development, highlight some notable examples of how such work has been conducted (and the particular contributions it has made), and offers some guidelines for those seeking to increase the sophistication and utility of qualitative methods in the evaluation of investment climate reforms.Publication Integrating Quantitative and Qualitative Research in Development Projects(Washington, DC: World Bank, 2000-06)"Much of the early work on poverty was highly quantitative:... It became increasingly clear, however, that while numbers are essential for policy and monitoring purposes, it is also important to understand people's perception of poverty and their mechanisms for coping with poverty and other situations of extreme economic and social stress." Researchers have recognized over the past few years that quantitative analysis of the incidence and trends in poverty, while essential for national economic development planning, must be complemented by qualitative methods that help planners and managers understand the cultural, social, political, and institutional context within which projects are designed and implemented. This report is based on a two-day workshop held in June 1998, where outside research specialists and World Bank staff discussed the importance of integrating these research methods. The participants reviewed experiences in the use of mixed-method approaches in Bank research and project design. This report is a result of those discussions. The report examines the need for integrated research approaches in social and economic development, presents case studies of integrated approaches in practice, and talks about lessons learned. Part I describes the evolution of interest in, and the potential benefits of integrated research, and, Part II presents case studies on how integrated approaches have been used in poverty analysis, education, health, and water supply and sanitation, while Part III discusses lessons learned with respect to the use of integrated approaches, and assesses the benefits that can be achieved.Publication Using Mixed Methods in Monitoring and Evaluation : Experiences from International Development(2010-03-01)This paper provides an overview of the various ways in which mixing qualitative and quantitative methods could add value to monitoring and evaluating development projects. In particular it examines how qualitative methods could address some of the limitations of randomized trials and other quantitative impact evaluation methods; it also explores the importance of examining "process" in addition to "impact", distinguishing design from implementation failures, and the value of mixed methods in the real-time monitoring of projects. It concludes by suggesting topics for future research -- including the use of mixed methods in constructing counterfactuals, and in conducting reasonable evaluations within severe time and budget constraints.
Users also downloaded
Showing related downloaded files
Publication Digital Progress and Trends Report 2023(Washington, DC: World Bank, 2024-03-05)Digitalization is the transformational opportunity of our time. The digital sector has become a powerhouse of innovation, economic growth, and job creation. Value added in the IT services sector grew at 8 percent annually during 2000–22, nearly twice as fast as the global economy. Employment growth in IT services reached 7 percent annually, six times higher than total employment growth. The diffusion and adoption of digital technologies are just as critical as their invention. Digital uptake has accelerated since the COVID-19 pandemic, with 1.5 billion new internet users added from 2018 to 2022. The share of firms investing in digital solutions around the world has more than doubled from 2020 to 2022. Low-income countries, vulnerable populations, and small firms, however, have been falling behind, while transformative digital innovations such as artificial intelligence (AI) have been accelerating in higher-income countries. Although more than 90 percent of the population in high-income countries was online in 2022, only one in four people in low-income countries used the internet, and the speed of their connection was typically only a small fraction of that in wealthier countries. As businesses in technologically advanced countries integrate generative AI into their products and services, less than half of the businesses in many low- and middle-income countries have an internet connection. The growing digital divide is exacerbating the poverty and productivity gaps between richer and poorer economies. The Digital Progress and Trends Report series will track global digitalization progress and highlight policy trends, debates, and implications for low- and middle-income countries. The series adds to the global efforts to study the progress and trends of digitalization in two main ways: · By compiling, curating, and analyzing data from diverse sources to present a comprehensive picture of digitalization in low- and middle-income countries, including in-depth analyses on understudied topics. · By developing insights on policy opportunities, challenges, and debates and reflecting the perspectives of various stakeholders and the World Bank’s operational experiences. This report, the first in the series, aims to inform evidence-based policy making and motivate action among internal and external audiences and stakeholders. The report will bring global attention to high-performing countries that have valuable experience to share as well as to areas where efforts will need to be redoubled.Publication The Container Port Performance Index 2023(Washington, DC: World Bank, 2024-07-18)The Container Port Performance Index (CPPI) measures the time container ships spend in port, making it an important point of reference for stakeholders in the global economy. These stakeholders include port authorities and operators, national governments, supranational organizations, development agencies, and other public and private players in trade and logistics. The index highlights where vessel time in container ports could be improved. Streamlining these processes would benefit all parties involved, including shipping lines, national governments, and consumers. This fourth edition of the CPPI relies on data from 405 container ports with at least 24 container ship port calls in the calendar year 2023. As in earlier editions of the CPPI, the ranking employs two different methodological approaches: an administrative (technical) approach and a statistical approach (using matrix factorization). Combining these two approaches ensures that the overall ranking of container ports reflects actual port performance as closely as possible while also being statistically robust. The CPPI methodology assesses the sequential steps of a container ship port call. ‘Total port hours’ refers to the total time elapsed from the moment a ship arrives at the port until the vessel leaves the berth after completing its cargo operations. The CPPI uses time as an indicator because time is very important to shipping lines, ports, and the entire logistics chain. However, time, as captured by the CPPI, is not the only way to measure port efficiency, so it does not tell the entire story of a port’s performance. Factors that can influence the time vessels spend in ports can be location-specific and under the port’s control (endogenous) or external and beyond the control of the port (exogenous). The CPPI measures time spent in container ports, strictly based on quantitative data only, which do not reveal the underlying factors or root causes of extended port times. A detailed port-specific diagnostic would be required to assess the contribution of underlying factors to the time a vessel spends in port. A very low ranking or a significant change in ranking may warrant special attention, for which the World Bank generally recommends a detailed diagnostic.Publication Global Economic Prospects, January 2024(Washington, DC: World Bank, 2024-01-09)Note: Chart 1.2.B has been updated on January 18, 2024. Chart 2.2.3 B has been updated on January 14, 2024. Global growth is expected to slow further this year, reflecting the lagged and ongoing effects of tight monetary policy to rein in inflation, restrictive credit conditions, and anemic global trade and investment. Downside risks include an escalation of the recent conflict in the Middle East, financial stress, persistent inflation, weaker-than-expected activity in China, trade fragmentation, and climate-related disasters. Against this backdrop, policy makers face enormous challenges. In emerging market and developing economies (EMDEs), commodity exporters face the enduring challenges posed by fiscal policy procyclicality and volatility, which highlight the need for robust fiscal frameworks. Across EMDEs, previous episodes of investment growth acceleration underscore the critical importance of macroeconomic and structural policies and an enabling institutional environment in bolstering investment and long-term growth. At the global level, cooperation needs to be strengthened to provide debt relief, facilitate trade integration, tackle climate change, and alleviate food insecurity.Publication Global Economic Prospects, June 2025(Washington, DC: World Bank, 2025-06-10)The global economy is facing another substantial headwind, emanating largely from an increase in trade tensions and heightened global policy uncertainty. For emerging market and developing economies (EMDEs), the ability to boost job creation and reduce extreme poverty has declined. Key downside risks include a further escalation of trade barriers and continued policy uncertainty. These challenges are exacerbated by subdued foreign direct investment into EMDEs. Global cooperation is needed to restore a more stable international trade environment and scale up support for vulnerable countries grappling with conflict, debt burdens, and climate change. Domestic policy action is also critical to contain inflation risks and strengthen fiscal resilience. To accelerate job creation and long-term growth, structural reforms must focus on raising institutional quality, attracting private investment, and strengthening human capital and labor markets. Countries in fragile and conflict situations face daunting development challenges that will require tailored domestic policy reforms and well-coordinated multilateral support.Publication Global Economic Prospects, January 2025(Washington, DC: World Bank, 2025-01-16)Global growth is expected to hold steady at 2.7 percent in 2025-26. However, the global economy appears to be settling at a low growth rate that will be insufficient to foster sustained economic development—with the possibility of further headwinds from heightened policy uncertainty and adverse trade policy shifts, geopolitical tensions, persistent inflation, and climate-related natural disasters. Against this backdrop, emerging market and developing economies are set to enter the second quarter of the twenty-first century with per capita incomes on a trajectory that implies substantially slower catch-up toward advanced-economy living standards than they previously experienced. Without course corrections, most low-income countries are unlikely to graduate to middle-income status by the middle of the century. Policy action at both global and national levels is needed to foster a more favorable external environment, enhance macroeconomic stability, reduce structural constraints, address the effects of climate change, and thus accelerate long-term growth and development.