Publication: Integrating Value for Money and Impact Evaluations: Issues, Institutions, and Opportunities
Loading...
Files in English
622 downloads
Date
2019-10
ISSN
Published
2019-10
Author(s)
Editor(s)
Abstract
This mixed methods study investigates why fewer than one in five impact evaluations integrates a value-for-money analysis of the development intervention being evaluated. This study distills four main insights from combined analysis of 33 semi-structured and unstructured interviews, surveys of 497 policy makers and 16 journal editors, and portfolio analyses of World Bank and worldwide impact evaluations. The study finds that low levels of training in cost data collection and analysis methods, together with a lack of standardization of the value-for-money assumptions (e.g., time horizons, discount rates, and economic or financial cost accounting) limit value-for-money integration into impact evaluations. Further eroding researchers' incentives, demand for cost evidence from the journals that publish impact evaluations is mixed. Ill-defined standards of rigor undermine editors' capacity to evaluate the quality of value-for-money analysis when it is integrated with impact evaluation evidence. Institutional funders of impact evaluations do not consistently demand that cost analysis be integrated into their funded evaluations. This study finds no evidence in support of the myth that policymakers do not demand cost evidence. Rather, it finds that researchers have few ways of knowing what kind of analysis policymakers need and when they need it. Improving the stock of impact evaluators who are cross trained in value-for-money methods, establishing standards in what constitutes rigor in costing, resolving methodological issues, and improving linkages between policymakers and researchers would lead to greater integration of value-for-money methods in impact evaluations.
Link to Data Set
Citation
“Brown, Elizabeth D.; Tanner, Jeffery C.. 2019. Integrating Value for Money and Impact Evaluations: Issues, Institutions, and Opportunities. Policy Research Working Paper;No. 9041. © World Bank. http://hdl.handle.net/10986/32586 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Publication The Macroeconomic Implications of Climate Change Impacts and Adaptation Options(Washington, DC: World Bank, 2025-05-29)Estimating the macroeconomic implications of climate change impacts and adaptation options is a topic of intense research. This paper presents a framework in the World Bank's macrostructural model to assess climate-related damages. This approach has been used in many Country Climate and Development Reports, a World Bank diagnostic that identifies priorities to ensure continued development in spite of climate change and climate policy objectives. The methodology captures a set of impact channels through which climate change affects the economy by (1) connecting a set of biophysical models to the macroeconomic model and (2) exploring a set of development and climate scenarios. The paper summarizes the results for five countries, highlighting the sources and magnitudes of their vulnerability --- with estimated gross domestic product losses in 2050 exceeding 10 percent of gross domestic product in some countries and scenarios, although only a small set of impact channels is included. The paper also presents estimates of the macroeconomic gains from sector-level adaptation interventions, considering their upfront costs and avoided climate impacts and finding significant net gross domestic product gains from adaptation opportunities identified in the Country Climate and Development Reports. Finally, the paper discusses the limits of current modeling approaches, and their complementarity with empirical approaches based on historical data series. The integrated modeling approach proposed in this paper can inform policymakers as they make proactive decisions on climate change adaptation and resilience.Publication Global Poverty Revisited Using 2021 PPPs and New Data on Consumption(Washington, DC: World Bank, 2025-06-05)Recent improvements in survey methodologies have increased measured consumption in many low- and lower-middle-income countries that now collect a more comprehensive measure of household consumption. Faced with such methodological changes, countries have frequently revised upward their national poverty lines to make them appropriate for the new measures of consumption. This in turn affects the World Bank’s global poverty lines when they are periodically revised. The international poverty line, which is based on the typical poverty line in low-income countries, increases by around 40 percent to $3.00 when the more recent national poverty lines as well as the 2021 purchasing power parities are incorporated. The net impact of the changes in international prices, the poverty line, and new survey data (including new data for India) is an increase in global extreme poverty by some 125 million people in 2022, and a significant shift of poverty away from South Asia and toward Sub-Saharan Africa. The changes at higher poverty lines, which are more relevant to middle-income countries, are mixed.Publication Geopolitical Fragmentation and Friendshoring(Washington, DC: World Bank, 2025-06-26)This paper examines the relationship between geopolitical fragmentation and friendshoring of foreign investments over time, countries, and sectors. The analysis uses comprehensive data on foreign direct investments covering greenfield projects, mergers and acquisitions, and stocks of affiliates, as well as data on four alternative measures of geopolitical distance between countries. The gravity estimations suggest that, first, geopolitical differences have a negative effect on foreign investments and the magnitude has heightened in the post-pandemic period compared to a decade ago. Second, it is primarily the companies from advanced Western economies whose foreign investment decisions are increasingly shaped by friendshoring forces. Finally, the paper shows that friendshoring is not only confined to strategic industries, implying that allocations of foreign direct investments may not solely reflect national security or resilience considerations.Publication Soaring Food Prices Threaten Recent Economic Gains in the EU(Washington, DC: World Bank, 2025-07-02)The surge in food prices following the 2021 economic rebound has become a significant concern for households, particularly low-income ones, in Bulgaria, Croatia, Poland, and Romania. Food price inflation, which surpasses general inflation rates, risks worsening poverty and food insecurity in these countries. This paper explores the distributional impacts of rising food prices and the effectiveness of government response measures. Low-income households, who allocate a larger share of their income to food, are disproportionately affected and are struggling to cope with unexpected expenses, leading to increased difficulties in accessing proper nutrition. Simulations indicate that rising food prices contribute to higher poverty rates and greater income inequality, especially among vulnerable populations. They also suggest that the main poverty-targeted social assistance schemes offer critical support for the extreme poor, but expanding both coverage and benefits is vital to shield all at-risk individuals. Targeted policies that balance immediate relief with long-term resilience-building are essential to addressing the challenges posed by escalating food prices.Publication Disentangling the Key Economic Channels through Which Infrastructure Affects Jobs(Washington, DC: World Bank, 2025-04-03)This paper takes stock of the literature on infrastructure and jobs published since the early 2000s, using a conceptual framework to identify the key channels through which different types of infrastructure impact jobs. Where relevant, it highlights the different approaches and findings in the cases of energy, digital, and transport infrastructure. Overall, the literature review provides strong evidence of infrastructure’s positive impact on employment, particularly for women. In the case of electricity, this impact arises from freeing time that would otherwise be spent on household tasks. Similarly, digital infrastructure, particularly mobile phone coverage, has demonstrated positive labor market effects, often driven by private sector investments rather than large public expenditures, which are typically required for other large-scale infrastructure projects. The evidence on structural transformation is also positive, with some notable exceptions, such as studies that find no significant impact on structural transformation in rural India in the cases of electricity and roads. Even with better market connections, remote areas may continue to lack economic opportunities, due to the absence of agglomeration economies and complementary inputs such as human capital. Accordingly, reducing transport costs alone may not be sufficient to drive economic transformation in rural areas. The spatial dimension of transformation is particularly relevant for transport, both internationally—by enhancing trade integration—and within countries, where economic development tends to drive firms and jobs toward urban centers, benefitting from economies scale and network effects. Turning to organizational transformation, evidence on skill bias in developing countries is more mixed than in developed countries and may vary considerably by context. Further research, especially on the possible reasons explaining the differences between developed and developing economies, is needed.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Georgia Public Expenditure Review : Diagnostics of Public Investment Management System(Washington, DC, 2014-06-11)Generating growth and creating jobs within a sustainable fiscal framework is Georgia s biggest macroeconomic challenge. Although Georgia registered rapid growth of 5.7 percent a year during 2010-13, unemployment remains high at 15 percent. New growth companies, especially in tourism and other service sectors, did not generate enough formal or even informal employment. Fiscal policy played a crucial role in Georgia s recent growth performance with a fiscal stimulus driven post-crisis recovery which increased deficit and debt levels followed by fiscal consolidation during 2010-12 when recovery took hold. The weak execution of the budget in 2013 and policy uncertainty were largely responsible for the growth slowdown during the year. Tackling the growth and jobs agenda in Georgia will require significant investment in human and physical capital and the government has a large role to play here. Additional spending, where it is needed, should be undertaken within the fiscal consolidation agenda of the government, designed to help restore the macroeconomic buffers needed to secure stability and sustain confidence in the future. The change in government in 2012 marked a shift in fiscal policy with prioritization of recurrent social expenditures over capital spending, thereby, increasing budget rigidity. During 2012-13, the government raised the benefit levels under the targeted social assistance (TSA) and pensions and introduced universal health care (UHC). As a result, the fiscal deficit is likely to increase from 2.6 percent of gross domestic product (GDP) in 2013 to 3.7 percent in 2014. Over the medium term, an aging population and the need to improve health outcomes and coverage of the poor in social assistance programs will keep social expenditures high at more than 9 percent of GDP. The share of capital expenditures will level off, meanwhile. Such an outcome will reduce the government s flexibility in trimming current expenditures in the future.Publication Romania - A Risk Analysis and Screening Approach for Climate Change Mitigation and Adaptation Options(Washington, DC, 2014-06)The current report is a milestone for preparing the Romanian climate change (CC) action plan. A CC action plan should be based on the consolidation and coordination of both mitigation and adaptation measures and options. An important step in the elaboration of such a plan is the selection of appropriate actions. This selection requires the involvement of competent authorities in the key sectors, including, in the case of Romania, transport, energy, urban, water, agriculture, and forestry. The screening approaches are used to facilitate a transparent decision-making process in various fields such as environmental impacts, dangerous substances, environmental footprint of products, or CC actions. The report is structured as follows: section one gives introduction. Section two summarizes the usage of screening approaches in the field of CC and environment by different European Union (EU) member states and by the European Commission. Section three discusses the main categories of mitigation and adaption measures that can be included in a CC action plan. Section four presents the screening approach and process and describes several complementary tools (multi-criteria analysis, cost-benefit analysis (CBA), cost effectiveness analysis (CEA), modeling) that should be used when the screening approach is insufficient. Section five provides a conclusion and key recommendations.Publication Economic Evaluation of Climate Change Adaptation Projects(Washington, DC, 2010)This paper identifies key challenges and solutions for carrying out project-level economic analysis of adaptation to climate change, both stand-alone and integrated into broader development projects. Very few projects addressing adaptation thus far have been subject to in-depth and rigorous economic analysis for a variety of reasons, including a lack of guidance on how to deal with assessments of the impacts of climate change, as well as with estimating costs and benefits of adaptation under uncertainty. The paper focus is on the agricultural sector, where the impacts of climate change have the potential to disrupt the livelihoods of rural populations in many regions and where adaptation must be given urgent consideration. Nevertheless, some of the approaches discussed are suitable to projects in other sectors as well. Finally, robust decision making (RDM) can provide an alternative quantitative decision analytic method that avoids subjective probability assessments and scenario predictions. RDM creates hundreds or thousands of plausible futures, in the judgment of the analyst, that are then used to systematically evaluate the performance of alternative actions. This approach facilitates identifying the set of conditions under which any particular alternative adaptation performs well or poorly, according to various evaluation criteria based on the decision maker's judgment. The decision maker can identify 'robust' alternatives that, compared to other alternatives, perform reasonably well across a wide range of plausible futures.Publication Cost-Effectiveness Measurement in Development : Accounting for Local Costs and Noisy Impacts(World Bank Group, Washington, DC, 2014-09)As evidence from rigorous impact evaluations grows in development, there have been more calls to complement impact evaluation analysis with cost analysis, so that policy makers can make investment decisions based on costs as well as impacts. This paper discusses important considerations for implementing cost-effectiveness analysis in the policy making process. The analysis is applied in the context of education interventions, although the findings generalize to other areas. First, the paper demonstrates a systematic method for characterizing the sensitivity of impact estimates. Second, the concept of context-specificity is applied to cost measurement: program costs vary greatly across contexts -- both within and across countries -- and with program complexity. The paper shows how adapting a single cost ingredient across settings dramatically shifts cost-effectiveness measures. Third, the paper provides evidence that interventions with fewer beneficiaries tend to have higher per-beneficiary costs, resulting in potential cost overestimates when extrapolating to large-scale applications. At the same time, recall bias may result in cost underestimates. The paper also discusses other challenges in measuring and extrapolating cost-effectiveness measures. For cost-effectiveness analysis to be useful, policy makers will require detailed, comparable, and timely cost reporting, as well as significant effort to ensure costs are relevant to the local environment.Publication Agreeing on Robust Decisions : New Processes for Decision Making under Deep Uncertainty(World Bank, Washington, DC, 2014-06)Investment decision making is already difficult for any diverse group of actors with different priorities and views. But the presence of deep uncertainties linked to climate change and other future conditions further challenges decision making by questioning the robustness of all purportedly optimal solutions. While decision makers can continue to use the decision metrics they have used in the past (such as net present value), alternative methodologies can improve decision processes, especially those that lead with analysis and end in agreement on decisions. Such "Agree-on-Decision" methods start by stress-testing options under a wide range of plausible conditions, without requiring us to agree ex ante on which conditions are more or less likely, and against a set of objectives or success metrics, without requiring us to agree ex ante on how to aggregate or weight them. As a result, these methods are easier to apply to contexts of large uncertainty or disagreement on values and objectives. This inverted process promotes consensus around better decisions and can help in managing uncertainty. Analyses performed in this way let decision makers make the decision and inform them on (1) the conditions under which an option or project is vulnerable; (2) the tradeoffs between robustness and cost, or between various objectives; and (3) the flexibility of various options to respond to changes in the future. In doing so, they put decision makers back in the driver's seat. A growing set of case studies shows that these methods can be applied in real-world contexts and do not need to be more costly or complicated than traditional approaches. Finally, while this paper focuses on climate change, a better treatment of uncertainties and disagreement would in general improve decision making and development outcomes.
Users also downloaded
Showing related downloaded files
Publication Tanzania(World Bank, Washington, DC, 2015-06)This study aims to achieve a better understanding of the agricultural risk and risk management situation in Tanzania with a view to identifying key solutions to reduce current gross domestic product (GDP) growth volatility. For the purpose of this assessment, risk is defined as the probability that an uncertain event will occur that can potentially produce losses to participants along the supply chain. Persistence of unmanaged risks in agriculture is a cause of great economic losses for farmers and other actors along the supply chains (for example, traders, processors, and exporters), affecting export earnings and food security. The agricultural sector risk assessment is a straightforward methodology based on a three-phase sequential process. Phase analyzes the chronological occurrence of inter-seasonal agricultural risks with a view to identify and prioritize the risks that are the drivers of agricultural GDP volatility. This report contains the findings and recommendations of the first phase and includes the identification, analysis, and prioritization of major risks facing the agricultural sector in Tanzania, as well as recommendations regarding key solutions. Chapter one gives introduction and context. Chapter two contains an overview of the agricultural sector and its performance, as well as a discussion of key agro-climatic, weather, and policy restrictions and opportunities. Chapter three includes an assessment of major risks (that is, production, market, and enabling environment risks) facing key export and food crops. Chapter four presents an estimate of historical losses due to realized production risks and a correlation of such losses with production volatility. Chapter five provides insights into the exposure to risks by different stakeholders and their actual capacities, vulnerabilities, and potential to manage agricultural risks. Chapter six presents a risk prioritization by different supply chains and discusses the possible solutions, as well as specific recommendations for the agricultural sector development program (ASDP).Publication Africa's Future, Africa's Challenge : Early Childhood Care and Development in Sub-Saharan Africa(Washington, DC : World Bank, 2008)This book seeks to achieve a balance, describing challenges that are being faced as well as developments that are underway. It seeks a balance in terms of the voices heard, including not just voices of the North commenting on the South, but voices from the South, and in concert with the North. It seeks to provide the voices of specialists and generalists, of those from international and local organizations, from academia and the field. It seeks a diversity of views and values. Such diversity and complexity are the reality of Sub-Saharan Africa (SSA) today. The major focus of this book is on SSA from the Sahel south. Approximately 130 million children between birth and age 6 live in SSA. Every year 27 million children are born, and every year 4.7 million children under age 5 die. Rates of birth and of child deaths are consistently higher in SSA than in any other part of the world; the under-5 mortality rate of 163 per 1,000 is twice that of the rest of the developing world and 30 times that of industrialized countries (UNICEF 2006). Of the children who are born, 65 percent will experience poverty, 14 million will be orphans affected by HIV/AIDS directly and within their families and one-third will experience exclusion because of their gender or ethnicity.Publication Ten Steps to a Results-Based Monitoring and Evaluation System : A Handbook for Development Practitioners(Washington, DC: World Bank, 2004)An effective state is essential to achieving socio-economic and sustainable development. With the advent of globalization, there are growing pressures on governments and organizations around the world to be more responsive to the demands of internal and external stakeholders for good governance, accountability and transparency, greater development effectiveness, and delivery of tangible results. Governments, parliaments, citizens, the private sector, Non-governmental Organizations (NGOs), civil society, international organizations, and donors are among the stakeholders interested in better performance. As demands for greater accountability and real results have increased, there is an attendant need for enhanced results-based monitoring and evaluation of policies, programs, and projects. This handbook provides a comprehensive ten-step model that will help guide development practitioners through the process of designing and building a results-based monitoring and evaluation system. These steps begin with a 'readiness assessment' and take the practitioner through the design, management, and importantly, the sustainability of such systems. The handbook describes each step in detail, the tasks needed to complete each one, and the tools available to help along the way.Publication World Development Report 2017(Washington, DC: World Bank, 2017-01-30)Why are carefully designed, sensible policies too often not adopted or implemented? When they are, why do they often fail to generate development outcomes such as security, growth, and equity? And why do some bad policies endure? This book addresses these fundamental questions, which are at the heart of development. Policy making and policy implementation do not occur in a vacuum. Rather, they take place in complex political and social settings, in which individuals and groups with unequal power interact within changing rules as they pursue conflicting interests. The process of these interactions is what this Report calls governance, and the space in which these interactions take place, the policy arena. The capacity of actors to commit and their willingness to cooperate and coordinate to achieve socially desirable goals are what matter for effectiveness. However, who bargains, who is excluded, and what barriers block entry to the policy arena determine the selection and implementation of policies and, consequently, their impact on development outcomes. Exclusion, capture, and clientelism are manifestations of power asymmetries that lead to failures to achieve security, growth, and equity. The distribution of power in society is partly determined by history. Yet, there is room for positive change. This Report reveals that governance can mitigate, even overcome, power asymmetries to bring about more effective policy interventions that achieve sustainable improvements in security, growth, and equity. This happens by shifting the incentives of those with power, reshaping their preferences in favor of good outcomes, and taking into account the interests of previously excluded participants. These changes can come about through bargains among elites and greater citizen engagement, as well as by international actors supporting rules that strengthen coalitions for reform.Publication World Development Report 1984(New York: Oxford University Press, 1984)Long-term needs and sustained effort are underlying themes in this year's report. As with most of its predecessors, it is divided into two parts. The first looks at economic performance, past and prospective. The second part is this year devoted to population - the causes and consequences of rapid population growth, its link to development, why it has slowed down in some developing countries. The two parts mirror each other: economic policy and performance in the next decade will matter for population growth in the developing countries for several decades beyond. Population policy and change in the rest of this century will set the terms for the whole of development strategy in the next. In both cases, policy changes will not yield immediate benefits, but delay will reduce the room for maneuver that policy makers will have in years to come.