Publication: The Rigor of Case-Based Causal Analysis: Busting Myths through a Demonstration
Loading...
Date
2023-05-02
ISSN
Published
2023-05-02
Author(s)
Editor(s)
Abstract
Several myths persist within research and evaluation circles about the power and limitations of evaluation designs that use cases (or case studies) as their primary empirical material (case-based evaluation designs). Using a real-world application, this paper busts two myths regarding the use of case-based designs in evaluations that aim to answer effectiveness questions and unpack the relationships between interventions and observed changes in outcomes (broadly known as causal analysis): that case studies cannot be used for causal analysis and that it is impossible to generalize from case studies. Through a detailed demonstration of how the evaluation of the World Bank’s support to carbon finance has been designed and implemented, the paper undoes these preconceived ideas about the inferential, explanatory, and generalizability power of case-based evaluation designs.
Link to Data Set
Citation
“Raimondo, Estelle. Vaessen, Jos; Hagh, Ariya, editors. 2023. The Rigor of Case-Based Causal Analysis: Busting Myths through a Demonstration. IEG Methods and Evaluation Capacity Development Working Paper Series. © World Bank. http://hdl.handle.net/10986/39773 License: CC BY-NC 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Aid and Reform in Africa : Lessons from Ten Case Studies(Washington, DC: World Bank, 2001-04)This book synthesizes the findings from ten case studies that investigate whether, when, and how foreign aid affected economic policy in Africa, and reveals the range of African policy experience. Results varied enormously, for example, while Ghana and Uganda were successful reformers that grew rapidly reducing poverty, Cote d'Ivoire and Ethiopia have shown significant reform recently, but its sustainability remains to be seen, and, in other countries, policies changed little, or even worsened. Based on the World Bank's Country Policy and Institutional Assessment, the study relates foreign aid in the 1990s, to a measure of overall economic policy, a broad measure that covers macroeconomic management, as well as effectiveness of the public sector in providing essential services for growth, and poverty reduction. In assessing aid, and reform policy, the study subdivides these countries in three groups: the post-socialist reformers (Ethiopia, Mali and Tanzania); the mixed reformers (Cote d'Ivoire, Kenya and Zambia), and the non-reformers (The Democratic Republic of Congo - Zaire - and Nigeria). Although defining "good policy", and how to measure it may be controversial, research and experience established a fair knowledge: absence of high inflation, functioning foreign exchange, openness to foreign trade, effective rule of law, and delivery of key services. Conclusions stipulate that key to successful reform, is a political movement for change; that key to beneficial aid is its disbursement alongside actual policy improvements; and, that technical assistance, and policy dialogue should continue a high level of finance in productive environments.Publication What Difference Does Good Monitoring and Evaluation Make to World Bank Project Performance?(World Bank, Washington, DC, 2016-06)For more than 20 years, the development community has claimed that monitoring and evaluation helps projects achieve their objectives. This study uses data from 1,300 World Bank projects evaluated between 2008 and 2014 to investigate this suggested link between the quality of monitoring and evaluation and project performance. The propensity score matching results indicate that the quality of monitoring and evaluation is significantly and positively associated with project outcome as institutionally measured at the World Bank. This positive relationship holds when controlling for project manager identity, and is robust to various specification choices. Through a systematic text analysis of the narrative produced by the Independent Evaluation Group to justify its monitoring and evaluation quality rating, the study shows that there are common markers of good quality monitoring and evaluation, such as: clear institutional setup and division of labor around monitoring and evaluation activities; simple monitoring and evaluation framework that is well aligned with clients' existing monitoring and evaluation systems; good integration with operational tasks; and a system that can generate regular and timely reporting, and that is used during and after lending.Publication Emerging Evidence on Vouchers and Faith-based Providers in Education : Case Studies from Africa, Latin America, and Asia(World Bank, 2009)The case studies in this book provide useful information on the characteristics of students and the performance of various types of schools that benefit from public-private partnerships. While these case studies are empirically grounded, their results are not necessarily of universal application, because context also matters. The authors are careful to point out that while one of the case studies is based on an experiment, the other case studies use instruments or matching methods that have their limitations. Yet a key result from this work is that sound analyses of existing data are feasible and can yield useful conclusions about the contribution that private service providers can offer to educational development. These case studies will encourage more researchers to undertake similar work to demonstrate the many options that developing countries have to reach their education goals.Publication Evaluating Social Funds : A Cross-Country Analysis of Community Investments(Washington, DC: World Bank, 2004)The study seeks to answer four questions that summarize the fundamental issues in the international debate about the capacity of social funds to improve beneficiaries' living conditions: o Do social funds reach poor areas and poor households? Do social funds deliver high-quality, sustainable investments? Do social funds affect living standards? How cost-efficient are social funds and the investments they finance, compared with other delivery mechanisms? The findings and lessons from this research reflect a specific moment in the evolution of six social funds and therefore may not fully predict the future impact of current investments. The evaluation assesses subprojects identified and implemented between 1993 and 1999, a period when longer-term objectives-such as increasing access to and utilization of basic services-began to supplant the funds' original emergency mandates. The time period selected allowed enough elapsed time following the implementation of the social fund subprojects to make measurement of impact and sustainability possible. The evaluation does not consider the effects of social fund projects on employment or on income generation-the original objectives of the first generation of social funds, which were introduced in Latin America. It also does not discuss the effect of social fund investments on capacity building-a more recent emphasis of social funds seeking to assist decentralization and community development.Publication Entrepreneurship Programs in Developing Countries : A Meta Regression Analysis(World Bank, Washington, DC, 2013-04)This paper provides a synthetic and systematic review on the effectiveness of various entrepreneurship programs in developing countries. It adopts a meta-regression analysis using 37 impact evaluation studies that were in the public domain by March 2012, and draws out several lessons on the design of the programs. The paper observes wide variation in program effectiveness across different interventions depending on outcomes, types of beneficiaries, and country context. Overall, entrepreneurship programs have a positive and large impact for youth and on business knowledge and practice, but no immediate translation into business set-up and expansion or increased income. At a disaggregate level by outcome groups, providing a package of training and financing is more effective for labor activities. In addition, financing support appears more effective for women and business training for existing entrepreneurs than other interventions to improve business performance.
Users also downloaded
Showing related downloaded files
Publication Remarks at the United Nations Biodiversity Conference(World Bank, Washington, DC, 2021-10-12)World Bank Group President David Malpass discussed biodiversity and climate change being closely interlinked, with terrestrial and marine ecosystems serving as critically important carbon sinks. At the same time climate change acts as a direct driver of biodiversity and ecosystem services loss. The World Bank has financed biodiversity conservation around the world, including over 116 million hectares of Marine and Coastal Protected Areas, 10 million hectares of Terrestrial Protected Areas, and over 300 protected habitats, biological buffer zones and reserves. The COVID pandemic, biodiversity loss, climate change are all reminders of how connected we are. The recovery from this pandemic is an opportunity to put in place more effective policies, institutions, and resources to address biodiversity loss.Publication Global Regulations, Institutional Development, and Market Authorities Perspective Toolkit (GRIDMAP) - Framework and Methodology(Washington, DC: World Bank, 2024-12-05)GRIDMAP--the Global Regulations, Institutional Development, and Market Authorities Perspective Toolkit--provides emerging markets and developing economies (EMDEs) with a “Minimum Package” of policies to build markets that are trustworthy, safe, and competitive. The “Minimum Package” sets out essential regulatory provisions, institutional arrangements, and implementation and enforcement needed for those markets to thrive. GRIDMAP will provide modules focused on various subjects of market regulation, such as consumer protection and data markets.Publication Education, Social Norms, and the Marriage Penalty(Washington, DC: World Bank, 2024-10-16)A growing literature attributes gender inequality in labor market outcomes in part to the reduction in female labor supply after childbirth, the child penalty. However, if social norms constrain married women’s activities outside the home, then marriage can independently reduce employment, even in the absence childbearing. Given the correlation in timing between childbirth and marriage, conventional estimates of child penalties will conflate these two effects. The paper studies the marriage penalty in South Asia, a context featuring conservative gender norms and low female labor force participation. The study introduces a split-sample, pseudo-panel approach that allows for the separation of marriage and child penalties even in the absence of individual-level panel data. Marriage reduces women’s labor force participation in South Asia by 12 percentage points, whereas the marginal penalty of childbearing is small. Consistent with the central roles of both opportunity costs and social norms, the marriage penalty is smaller among cohorts with higher education and less conservative gender attitudes.Publication Economic Recovery(World Bank, Washington, DC, 2021-04-06)World Bank Group President David Malpass spoke about the world facing major challenges, including COVID, climate change, rising poverty and inequality and growing fragility and violence in many countries. He highlighted vaccines, working closely with Gavi, WHO, and UNICEF, the World Bank has conducted over one hundred capacity assessments, many even more before vaccines were available. The World Bank Group worked to achieve a debt service suspension initiative and increased transparency in debt contracts at developing countries. The World Bank Group is finalizing a new climate change action plan, which includes a big step up in financing, building on their record climate financing over the past two years. He noted big challenges to bring all together to achieve GRID: green, resilient, and inclusive development. Janet Yellen, U.S. Secretary of the Treasury, mentioned focusing on vulnerable people during the pandemic. Kristalina Georgieva, Managing Director of the International Monetary Fund, focused on giving everyone a fair shot during a sustainable recovery. All three commented on the importance of tackling climate change.Publication Media and Messages for Nutrition and Health(World Bank, Washington, DC, 2020-06)The Lao People’s Democratic Republic (Lao PDR) has experienced rapid and significant economic growth over the past decade. However, poor nutritional outcomes remain a concern. Rates of childhood undernutrition are particularly high in remote, rural, and upland areas. Media have the potential to play an important role in shaping health and nutrition–related behaviors and practices as well as in promoting sociocultural and economic development that might contribute to improved nutritional outcomes. This report presents the results of a media audit (MA) that was conducted to inform the development and production of mass media advocacy and communication strategies and materials with a focus on maternal and child health and nutrition that would reach the most people from the poorest communities in northern Lao PDR. Making more people aware of useful information, essential services and products and influencing them to use these effectively is the ultimate goal of mass media campaigns, and the MA measures the potential effectiveness of media efforts to reach this goal. The effectiveness of communication channels to deliver health and nutrition messages to target beneficiaries to ensure maximum reach and uptake can be viewed in terms of preferences, satisfaction, and trust. Overall, the four most accessed media channels for receiving information among communities in the study areas were village announcements, mobile phones, television, and out-of-home (OOH) media. Of the accessed media channels, the top three most preferred channels were village announcements (40 percent), television (26 percent), and mobile phones (19 percent). In terms of trust, village announcements were the most trusted source of information (64 percent), followed by mobile phones (14 percent) and television (11 percent). Hence of all the media channels, village announcements are the most preferred, have the most satisfied users, and are the most trusted source of information in study communities from four provinces in Lao PDR with some of the highest burden of childhood undernutrition.