Publication: Learning from the Experiments that Never Happened : Lessons from Trying to Conduct Randomized Evaluations of Matching Grant Programs in Africa
Loading...
Files in English
400 downloads
Published
2014-01-08
ISSN
0889-1583
Date
2014-05-06
Author(s)
Editor(s)
Abstract
Matching grants are one of the most common policy instruments used by developing country governments to try to foster technological upgrading, innovation, exports, use of business development services and other activities leading to firm growth. However, since they involve subsidizing firms, the risk is that they could crowd out private investment, subsidizing activities that firms were planning to undertake anyway, or lead to pure private gains, rather than generating the public gains that justify government intervention. As a result, rigorous evaluation of the effects of such programs is important. We attempted to implement randomized experiments to evaluate the impact of seven matching grant programs offered in six African countries, but in each case we were unable to complete an experimental evaluation. One critique of development research is publication bias, whereby only “interesting” results get published. Our hope is to mitigate this bias by learning from the experiments that never happened. We describe the three main proximate reasons for lack of implementation: continued project delays, politicians not willing to allow random assignment, and low program take-up; and then delve into the underlying causes of these occurring. Political economy, overly stringent eligibility criteria that do not take account of where value-added may be highest, a lack of attention to detail in “last mile” issues, incentives facing project implementation staff, and the way impact evaluations are funded, and all help explain the failure of randomization. We draw lessons from these experiences for both the implementation and the possible evaluation of future projects.
Link to Data Set
Digital Object Identifier
Associated content
Other publications in this report series
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication Learning from the Experiments That Never Happened : Lessons from Trying to Conduct Randomized Evaluations of Matching Grant Programs in Africa(World Bank, Washington, DC, 2012-12)Matching grants are one of the most common policy instruments used by developing country governments to try to foster technological upgrading, innovation, exports, use of business development services and other activities leading to firm growth. However, since they involve subsidizing firms, the risk is that they could crowd out private investment, subsidizing activities that firms were planning to undertake anyway, or lead to pure private gains, rather than generating the public gains that justify government intervention. As a result, rigorous evaluation of the effects of such programs is important. The authors attempted to implement randomized experiments to evaluate the impact of seven matching grant programs offered in six African countries, but in each case were unable to complete an experimental evaluation. One critique of randomized experiments is publication bias, whereby only those experiments with "interesting" results get published. The hope is to mitigate this bias by learning from the experiments that never happened. This paper describes the three main proximate reasons for lack of implementation: continued project delays, politicians not willing to allow random assignment, and low program take-up; and then delves into the underlying causes of these occurring. Political economy, overly stringent eligibility criteria that do not take account of where value-added may be highest, a lack of attention to detail in "last mile" issues, incentives facing project implementation staff, and the way impact evaluations are funded, and all help explain the failure of randomization. Lessons are drawn from these experiences for both the implementation and the possible evaluation of future projects.Publication Lessons for Matching Grant Programs from Failed Attempts to Evaluate Them(World Bank, Washington, DC, 2013-01)This note illustrates a case where attempting to do impact evaluations can provide valuable knowledge, even if the impact evaluations ultimately are unable to be completed.Publication How Should the Government Bring Small Firms into the Formal System? Experimental Evidence from Malawi(World Bank, Washington, DC, 2018-10)Developing country governments seek to reduce the pervasive informality of firms for multiple reasons: increasing the tax base, helping firms access formal markets and grow, increasing the rule of law, and as a means to obtain data that can be used for other government functions. However, there is debate as to the best approach for achieving these goals. This study conducted a randomized experiment in Malawi to test three alternatives: (a) assisting firms to obtain a business registration certificate that offers access to formal markets but imposes no tax obligations; (b) assisting firms to obtain business registration and tax registration; and (c) supplementing the assistance to obtain business registration with a bank information session intended to help firms utilize one of the key potential benefits of formalizing. The study finds incredibly high demand for obtaining a formal status that is separate from tax obligations, and very low take-up of tax registration. Business registration alone has no impact on access to formal markets or firm performance. However, coupling registration assistance with the bank information session increases the use of formal financial services, and results in increases in firm sales by 20 percent and profits by 15 percent. The results highlight the advantages of separating business and tax registration, but also the need to assist firms in benefiting from their new formal status.Publication How Can Matching Grants in Agriculture Facilitate Access to Finance? Lessons Learned from World Bank Group’s Experience(World Bank, Washington, DC, 2019)Matching grants are an instrument aimed at promoting private sector development that have been used extensively over the past years, in particular for agriculture development. this paper focuses on three specific issues: (1) What is the rationale for using matching grants in agriculture and why does the financial sector matter? (2) What has been the specific experience with WBG matching grants for agriculture and what are the key drivers of success? (3) What are the various models of linkages with financial institutions and how can matching grants be used to promote financial inclusion? Based on this analysis, this paper suggests emerging good practice on when to use matching grants for agriculture, and how to design them in a way that promotes sustainable impact and linkages with the financial sector.Publication Testing the Importance of Search Frictions, Matching, and Reservation Prestige through Randomized Experiments in Jordan(World Bank Group, Washington, DC, 2014-09-01)Unemployment rates for tertiary-educated youth in Jordan are high, as is the duration of unemployment. Two randomized experiments in Jordan were used to test different theories that may explain this phenomenon. The first experiment tested the role of search and matching frictions by providing firms and job candidates with an intensive screening and matching service based on educational backgrounds and psychometric assessments. Although more than 1,000 matches were made, youth rejected the opportunity to even have an interview in 28 percent of cases, and when a job offer was received, they rejected this offer or quickly quit the job 83 percent of the time. A second experiment built on the first by examining the willingness of educated, unemployed youth to apply for jobs of varying levels of prestige. Youth applied to only a small proportion of the job openings they were told about, with application rates higher for higher prestige jobs than lower prestige jobs. Youth failed to show up for the majority of interviews scheduled for low prestige jobs. The results suggest that reservation prestige is an important factor underlying the unemployment of educated Jordanian youth.
Users also downloaded
Showing related downloaded files
Publication The Container Port Performance Index 2020 to 2024: Trends and Lessons Learned(Washington, DC: World Bank, 2025-09-22)The Container Port Performance Index (CPPI) provides a global benchmark of how container ports perform in handling vessel calls. Developed jointly by the World Bank and S&P Global Market Intelligence, it measures the time ships spend in port and relates this to the number of containers moved during that time. This approach makes the CPPI a unique diagnostic tool that can highlight patterns in port operations and shed light on global and regional supply chain dynamics. Now in its fifth edition, the CPPI report covers the period from 2020 to 2024. It builds on a well-established methodology to generate scores for more than 400 container ports worldwide. Over time, the CPPI has become a trusted reference point for policymakers, industry stakeholders, and researchers who seek to understand how ports adapt to shocks, recover from disruptions, and identify opportunities for investments, reform and modernization. A major innovation in this edition is the introduction of multi-year trend analysis. Rather than presenting annual snapshots, the report now tracks how CPPI scores have changed across five years. This longitudinal perspective reveals shifts in port performance, showing where scores have risen, fallen, or remained stable. By linking these movements to external factors, the CPPI offers insights into how global and regional supply chains evolve under pressure. The results clearly mirror the crises that have shaken global trade. During the COVID-19 pandemic, CPPI scores in different regions declined sharply as congestion, equipment shortages, and delays overwhelmed many ports. By 2023, global averages rebounded in parallel with easing freight markets and reduced congestion. Yet 2024 brought new challenges: the Red Sea crisis disrupted major trade lanes, while climate-related constraints at the Panama Canal added further stress. These shocks were reflected in lower global and several regional average scores, underscoring the vulnerability of maritime transport to geopolitical and environmental events. The CPPI is not about comparing one port against another, but about understanding changes in performance over time. Ports that improved their scores often did so by reducing time at anchor, optimizing berth operations, investing in digital tools, and strengthening coordination across logistics partners. The evidence confirms that improvements are possible across ports of all sizes, and that rising scores are linked to deliberate actions to minimize time in port relative to containers moved. By consolidating five years of results, this edition transforms the CPPI into a long-term reference point. It shows how global crises have affected shipping, how different regions have adapted, and what lessons can be drawn for future resilience. The World Bank and S&P Global Market Intelligence remain committed to maintaining the CPPI as a global public good, providing transparency, comparability, and practical insights to support more reliable and sustainable maritime supply chains.Publication Global Economic Prospects, June 2025(Washington, DC: World Bank, 2025-06-10)The global economy is facing another substantial headwind, emanating largely from an increase in trade tensions and heightened global policy uncertainty. For emerging market and developing economies (EMDEs), the ability to boost job creation and reduce extreme poverty has declined. Key downside risks include a further escalation of trade barriers and continued policy uncertainty. These challenges are exacerbated by subdued foreign direct investment into EMDEs. Global cooperation is needed to restore a more stable international trade environment and scale up support for vulnerable countries grappling with conflict, debt burdens, and climate change. Domestic policy action is also critical to contain inflation risks and strengthen fiscal resilience. To accelerate job creation and long-term growth, structural reforms must focus on raising institutional quality, attracting private investment, and strengthening human capital and labor markets. Countries in fragile and conflict situations face daunting development challenges that will require tailored domestic policy reforms and well-coordinated multilateral support.Publication Digital Progress and Trends Report 2023(Washington, DC: World Bank, 2024-03-05)Digitalization is the transformational opportunity of our time. The digital sector has become a powerhouse of innovation, economic growth, and job creation. Value added in the IT services sector grew at 8 percent annually during 2000–22, nearly twice as fast as the global economy. Employment growth in IT services reached 7 percent annually, six times higher than total employment growth. The diffusion and adoption of digital technologies are just as critical as their invention. Digital uptake has accelerated since the COVID-19 pandemic, with 1.5 billion new internet users added from 2018 to 2022. The share of firms investing in digital solutions around the world has more than doubled from 2020 to 2022. Low-income countries, vulnerable populations, and small firms, however, have been falling behind, while transformative digital innovations such as artificial intelligence (AI) have been accelerating in higher-income countries. Although more than 90 percent of the population in high-income countries was online in 2022, only one in four people in low-income countries used the internet, and the speed of their connection was typically only a small fraction of that in wealthier countries. As businesses in technologically advanced countries integrate generative AI into their products and services, less than half of the businesses in many low- and middle-income countries have an internet connection. The growing digital divide is exacerbating the poverty and productivity gaps between richer and poorer economies. The Digital Progress and Trends Report series will track global digitalization progress and highlight policy trends, debates, and implications for low- and middle-income countries. The series adds to the global efforts to study the progress and trends of digitalization in two main ways: · By compiling, curating, and analyzing data from diverse sources to present a comprehensive picture of digitalization in low- and middle-income countries, including in-depth analyses on understudied topics. · By developing insights on policy opportunities, challenges, and debates and reflecting the perspectives of various stakeholders and the World Bank’s operational experiences. This report, the first in the series, aims to inform evidence-based policy making and motivate action among internal and external audiences and stakeholders. The report will bring global attention to high-performing countries that have valuable experience to share as well as to areas where efforts will need to be redoubled.Publication Global Economic Prospects, January 2025(Washington, DC: World Bank, 2025-01-16)Global growth is expected to hold steady at 2.7 percent in 2025-26. However, the global economy appears to be settling at a low growth rate that will be insufficient to foster sustained economic development—with the possibility of further headwinds from heightened policy uncertainty and adverse trade policy shifts, geopolitical tensions, persistent inflation, and climate-related natural disasters. Against this backdrop, emerging market and developing economies are set to enter the second quarter of the twenty-first century with per capita incomes on a trajectory that implies substantially slower catch-up toward advanced-economy living standards than they previously experienced. Without course corrections, most low-income countries are unlikely to graduate to middle-income status by the middle of the century. Policy action at both global and national levels is needed to foster a more favorable external environment, enhance macroeconomic stability, reduce structural constraints, address the effects of climate change, and thus accelerate long-term growth and development.Publication The Container Port Performance Index 2023(Washington, DC: World Bank, 2024-07-18)The Container Port Performance Index (CPPI) measures the time container ships spend in port, making it an important point of reference for stakeholders in the global economy. These stakeholders include port authorities and operators, national governments, supranational organizations, development agencies, and other public and private players in trade and logistics. The index highlights where vessel time in container ports could be improved. Streamlining these processes would benefit all parties involved, including shipping lines, national governments, and consumers. This fourth edition of the CPPI relies on data from 405 container ports with at least 24 container ship port calls in the calendar year 2023. As in earlier editions of the CPPI, the ranking employs two different methodological approaches: an administrative (technical) approach and a statistical approach (using matrix factorization). Combining these two approaches ensures that the overall ranking of container ports reflects actual port performance as closely as possible while also being statistically robust. The CPPI methodology assesses the sequential steps of a container ship port call. ‘Total port hours’ refers to the total time elapsed from the moment a ship arrives at the port until the vessel leaves the berth after completing its cargo operations. The CPPI uses time as an indicator because time is very important to shipping lines, ports, and the entire logistics chain. However, time, as captured by the CPPI, is not the only way to measure port efficiency, so it does not tell the entire story of a port’s performance. Factors that can influence the time vessels spend in ports can be location-specific and under the port’s control (endogenous) or external and beyond the control of the port (exogenous). The CPPI measures time spent in container ports, strictly based on quantitative data only, which do not reveal the underlying factors or root causes of extended port times. A detailed port-specific diagnostic would be required to assess the contribution of underlying factors to the time a vessel spends in port. A very low ranking or a significant change in ranking may warrant special attention, for which the World Bank generally recommends a detailed diagnostic.