Publication: On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics
Loading...
Date
2016-07
ISSN
Published
2016-07
Author(s)
Editor(s)
Abstract
Estimation of empirical relationships is prone to bias. Economists have carefully studied sources of bias in structural and quasi-experimental approaches, but the randomized control trial (RCT) has only begun to receive such scrutiny. This paper argues that several lessons from medicine, derived from analysis of thousands of RCTs establishing a clear link between certain practices and biased estimates, can be used to reduce the risk of bias in economics RCTs. It identifies the subset of these lessons applicable to economics and uses them to assess risk of bias in estimates from economics RCTs published between 2001 and 2011. In comparison to medical studies, most economics studies examined do not report important details on study design necessary to assess risk of bias. Many report practices that suggest risk of bias, though this does not necessarily mean bias resulted. The paper concludes with suggestions on how to remedy these issues.
Link to Data Set
Citation
“Eble, Alex; Boone, Peter; Elbourne, Diana. 2016. On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics. Policy Research Working Paper;No. 7746. © World Bank. http://hdl.handle.net/10986/24828 License: CC BY 3.0 IGO.”
Associated URLs
Associated content
Other publications in this report series
Publication The Future of Poverty(Washington, DC: World Bank, 2025-07-15)Climate change is increasingly acknowledged as a critical issue with far-reaching socioeconomic implications that extend well beyond environmental concerns. Among the most pressing challenges is its impact on global poverty. This paper projects the potential impacts of unmitigated climate change on global poverty rates between 2023 and 2050. Building on a study that provided a detailed analysis of how temperature changes affect economic productivity, this paper integrates those findings with binned data from 217 countries, sourced from the World Bank’s Poverty and Inequality Platform. By simulating poverty rates and the number of poor under two climate change scenarios, the paper uncovers some alarming trends. One of the primary findings is that the number of people living in extreme poverty worldwide could be nearly doubled due to climate change. In all scenarios, Sub-Saharan Africa is projected to bear the brunt, contributing the largest number of poor people, with estimates ranging between 40.5 million and 73.5 million by 2050. Another significant finding is the disproportionate impact of inequality on poverty. Even small increases in inequality can lead to substantial rises in poverty levels. For instance, if every country’s Gini coefficient increases by just 1 percent between 2022 and 2050, an additional 8.8 million people could be pushed below the international poverty line by 2050. In a more extreme scenario, where every country’s Gini coefficient increases by 10 percent between 2022 and 2050, the number of people falling into poverty could rise by an additional 148.8 million relative to the baseline scenario. These findings underscore the urgent need for comprehensive climate policies that not only mitigate environmental impacts but also address socioeconomic vulnerabilities.Publication Central Bank Independence and Sovereign Borrowing(Washington, DC: World Bank, 2025-07-25)This paper studies the impact of central bank independence on sovereign borrowing, using an index that captures institutional constraints on central bank lending to the government across 155 countries from 1972 to 2023. The findings show that tighter lending to the executive significantly reduces sovereign interest rates and raises the debt-to-gross domestic product ratio in developing countries. These effects reflect the executive’s improved ability to borrow at lower costs under greater central bank independence. The results are robust to multiple tests, but there are no significant effects in advanced economies. From a policy perspective, the results highlight the key role of independent central banks as catalysts for reducing governments’ borrowing costs and enhancing the government’s borrowing capacity.Publication Disentangling the Key Economic Channels through Which Infrastructure Affects Jobs(Washington, DC: World Bank, 2025-04-03)This paper takes stock of the literature on infrastructure and jobs published since the early 2000s, using a conceptual framework to identify the key channels through which different types of infrastructure impact jobs. Where relevant, it highlights the different approaches and findings in the cases of energy, digital, and transport infrastructure. Overall, the literature review provides strong evidence of infrastructure’s positive impact on employment, particularly for women. In the case of electricity, this impact arises from freeing time that would otherwise be spent on household tasks. Similarly, digital infrastructure, particularly mobile phone coverage, has demonstrated positive labor market effects, often driven by private sector investments rather than large public expenditures, which are typically required for other large-scale infrastructure projects. The evidence on structural transformation is also positive, with some notable exceptions, such as studies that find no significant impact on structural transformation in rural India in the cases of electricity and roads. Even with better market connections, remote areas may continue to lack economic opportunities, due to the absence of agglomeration economies and complementary inputs such as human capital. Accordingly, reducing transport costs alone may not be sufficient to drive economic transformation in rural areas. The spatial dimension of transformation is particularly relevant for transport, both internationally—by enhancing trade integration—and within countries, where economic development tends to drive firms and jobs toward urban centers, benefitting from economies scale and network effects. Turning to organizational transformation, evidence on skill bias in developing countries is more mixed than in developed countries and may vary considerably by context. Further research, especially on the possible reasons explaining the differences between developed and developing economies, is needed.Publication Crowding Out and Banking Crises(Washington, DC: World Bank, 2025-07-22)This paper studies the effect of government issuance on firm issuance during banking crises using transaction-level bond and loan data from 66 countries between 1991 and 2017. Governments rarely issue loans, preferring to issue in bond markets. In contrast, firms receive most of their financing from banks. During banking crises, as the supply of domestic loans decreases, firms switch to issuing bonds in domestic markets. The paper uses a novel instrument based on maturing debt to overcome the potential endogeneity of government issuance. The findings show that firms must compete with the government for funds in the domestic bond market and are crowded out from this market as a result. This happens not only in developing countries, but in advanced countries as well. The paper also shows that firms with the ability to tap international debt markets switch to these markets when crowding out occurs in domestic bond markets. Lastly, the paper shows that more developed domestic bond markets mitigate, but do not eliminate, the degree to which crowding out occurs.Publication Designing and Analyzing Powerful Experiments(Washington, DC: World Bank, 2025-07-22)This paper offers practical advice on how to improve statistical power in randomized experiments through choices and actions researchers can take at the design, implementation, and analysis stages. At the design stage, the choice of estimand, choice of treatment, and decisions that affect the residual variance and intra-cluster correlation can all affect power for a given sample size. At the implementation stage, researchers can boost power through increasing compliance with treatment, reducing attrition, and improving outcome measurement. At the analysis stage, power can be increased through using different test statistics or estimands, through the choice of control variables, and through incorporating informative priors in a Bayesian analysis. A key message is that it does not make sense to talk of “the” power of an experiment. A study can be well-powered for one outcome or estimand, but not others, and a fixed sample size can yield very different levels of power depending on researcher decisions.
Journal
Journal Volume
Journal Issue
Collections
Related items
Showing items related by metadata.
Publication On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics(Published by Oxford University Press on behalf of the World Bank, 2017-10-01)Estimation of empirical relationships is prone to bias. Economists have carefully studied sources of bias in structural and quasi-experimental approaches, but the randomized control trial (RCT) has only begun to receive such scrutiny. In this paper, we argue that several lessons from medicine, derived from analysis of thousands of RCTs establishing a clear link between certain practices and biased estimates, can be used to reduce the risk of bias in economics RCTs. We identify the subset of these lessons applicable to economics and use them to assess risk of bias in estimates from economics RCTs published between 2001 and 2011. In comparison to medical studies, we find most economics studies do not report important details on study design necessary to assess risk of bias. Many report practices that suggest risk of bias, though this does not necessarily mean bias resulted. We conclude with suggestions on how to remedy these issues.Publication Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity(Published by Oxford University Press on behalf of the World Bank, 2018-02-01)When properly implemented, Randomized Controlled Trials (RCT) achieve a high degree of internal validity. Yet, if an RCT is to inform policy, it is critical to establish external validity. This paper systematically reviews all RCTs conducted in developing countries and published in leading economic journals between 2009 and 2014 with respect to how they deal with external validity. Following Duflo, Glennerster, and Kremer (2008), we scrutinize the following hazards to external validity: Hawthorne effects, general equilibrium effects, specific sample problems, and special care in treatment provision. Based on a set of objective indicators, we find that the majority of published RCTs does not discuss these hazards and many do not provide the necessary information to assess potential problems. The paper calls for including external validity dimensions in a more systematic reporting on the results of RCTs. This may create incentives to avoid overgeneralizing findings and help policy makers to interpret results appropriately.Publication Evaluation of Development Programs : Randomized Controlled Trials or Regressions?(World Bank, Washington, DC, 2013-09)Can project evaluation methods be used to evaluate programs: complex interventions involving multiple activities? A program evaluation cannot be based simply on separate evaluations of its components if interactions between the activities are important. In this paper a measure is proposed, the total program effect (TPE), which is an extension of the average treatment effect on the treated (ATET). It explicitly takes into account that in the real world (with heterogeneous treatment effects) individual treatment effects and program assignment are often correlated. The TPE can also deal with the common situation in which such a correlation is the result of decisions on (intended) program participation not being taken centrally. In this context RCTs are less suitable even for the simplest interventions. The TPE can be estimated by applying regression techniques to observational data from a representative sample from the targeted population. The approach is illustrated with an evaluation of a health insurance program in Vietnam.Publication Do Sanitation Improvements Reduce Fecal Contamination of Water, Hands, Food, Soil, and Flies? Evidence from a Cluster-Randomized Controlled Trial in Rural Bangladesh(American Chemical Society, 2018-09-26)Sanitation improvements have had limited effectiveness in reducing the spread of fecal pathogens into the environment. We conducted environmental measurements within a randomized controlled trial in Bangladesh that implemented individual and combined water treatment, sanitation, handwashing (WSH) and nutrition interventions (WASH Benefits, NCT01590095). Following approximately 4 months of intervention, we enrolled households in the trial’s control, sanitation and combined WSH arms to assess whether sanitation improvements, alone and coupled with water treatment and handwashing, reduce fecal contamination in the domestic environment. We quantified fecal indicator bacteria in samples of drinking and ambient waters, child hands, food given to young children, courtyard soil and flies. In the WSH arm, Escherichia coli prevalence in stored drinking water was reduced by 62% (prevalence ratio = 0.38 (0.32, 0.44)) and E. coli concentration by 1-log (Δlog10 = −0.88 (−1.01, −0.75)). The interventions did not reduce E. coli along other sampled pathways. Ambient contamination remained high among intervention households. Potential reasons include noncommunity-level sanitation coverage, child open defecation, animal fecal sources, or naturalized E. coli in the environment. Future studies should explore potential threshold effects of different levels of community sanitation coverage on environmental contamination.Publication A Randomized Control Trial of a Peer Adherence and Nutritional Support Program for Public Sector Antiretroviral Patients(World Bank, Washington, DC, 2016-07)Access to antiretroviral treatment has expanded rapidly in South Africa, making it the country in the world with the largest treatment program. As antiretroviral treatment coverage continues to rise in resource-constrained settings, effective community-based adherence support interventions are of central importance in ensuring the long-term sustainability of treatment. This paper reports the findings from a randomized control trial of a peer adherence and nutritional support program implemented in a public health care setting in South Africa's antiretroviral treatment program. The analysis assesses the impact of these peer adherence and nutritional support interventions on self-reported adherence, timeliness of clinic and hospital visits, and immunologic response to antiretroviral treatment. Peer adherence and nutritional support improved the timeliness of adults´ clinic and hospital visits for routine follow-up while on antiretroviral treatment. Peer adherence support impacted positively on immunologic response to antiretroviral treatment. Scale-up of effective and sustainable community-based, peer-driven adherence and nutritional support interventions should form part of the United Nations AIDS Treatment 2.0 strategy's community mobilization and health system strengthening pillar.
Users also downloaded
Showing related downloaded files
Publication Classroom Assessment to Support Foundational Literacy(Washington, DC: World Bank, 2025-03-21)This document focuses primarily on how classroom assessment activities can measure students’ literacy skills as they progress along a learning trajectory towards reading fluently and with comprehension by the end of primary school grades. The document addresses considerations regarding the design and implementation of early grade reading classroom assessment, provides examples of assessment activities from a variety of countries and contexts, and discusses the importance of incorporating classroom assessment practices into teacher training and professional development opportunities for teachers. The structure of the document is as follows. The first section presents definitions and addresses basic questions on classroom assessment. Section 2 covers the intersection between assessment and early grade reading by discussing how learning assessment can measure early grade reading skills following the reading learning trajectory. Section 3 compares some of the most common early grade literacy assessment tools with respect to the early grade reading skills and developmental phases. Section 4 of the document addresses teacher training considerations in developing, scoring, and using early grade reading assessment. Additional issues in assessing reading skills in the classroom and using assessment results to improve teaching and learning are reviewed in section 5. Throughout the document, country cases are presented to demonstrate how assessment activities can be implemented in the classroom in different contexts.Publication Global Economic Prospects, June 2024(Washington, DC: World Bank, 2024-06-11)After several years of negative shocks, global growth is expected to hold steady in 2024 and then edge up in the next couple of years, in part aided by cautious monetary policy easing as inflation gradually declines. However, economic prospects are envisaged to remain tepid, especially in the most vulnerable countries. Risks to the outlook, while more balanced, are still tilted to the downside, including the possibility of escalating geopolitical tensions, further trade fragmentation, and higher-for-longer interest rates. Natural disasters related to climate change could also hinder activity. Subdued growth prospects across many emerging market and developing economies and continued risks underscore the need for decisive policy action at the global and national levels. Global Economic Prospects is a World Bank Group Flagship Report that examines global economic developments and prospects, with a special focus on emerging market and developing economies, on a semiannual basis (in January and June). Each edition includes analytical pieces on topical policy challenges faced by these economies.Publication Business Ready 2024(Washington, DC: World Bank, 2024-10-03)Business Ready (B-READY) is a new World Bank Group corporate flagship report that evaluates the business and investment climate worldwide. It replaces and improves upon the Doing Business project. B-READY provides a comprehensive data set and description of the factors that strengthen the private sector, not only by advancing the interests of individual firms but also by elevating the interests of workers, consumers, potential new enterprises, and the natural environment. This 2024 report introduces a new analytical framework that benchmarks economies based on three pillars: Regulatory Framework, Public Services, and Operational Efficiency. The analysis centers on 10 topics essential for private sector development that correspond to various stages of the life cycle of a firm. The report also offers insights into three cross-cutting themes that are relevant for modern economies: digital adoption, environmental sustainability, and gender. B-READY draws on a robust data collection process that includes specially tailored expert questionnaires and firm-level surveys. The 2024 report, which covers 50 economies, serves as the first in a series that will expand in geographical coverage and refine its methodology over time, supporting reform advocacy, policy guidance, and further analysis and research.Publication MIGA Annual Report 2021(Washington, DC: Multilateral Investment Guarantee Agency, 2021-10-01)In FY21, MIGA issued 5.2 billion US Dollars in new guarantees across 40 projects. These projects are expected to provide 784,000 people with new or improved electricity service, create over 14,000 jobs, generate over 362 million US Dollars in taxes for the host countries, and enable about 1.3 billion US Dollars in loans to businesses—critical as countries around the world work to keep their economies afloat. Of the 40 projects supported during FY21, 85 percent addressed at least one of the strategic priority areas, namely, IDA-eligible countries (lower-income), fragile and conflict affected situations (FCS), and climate finance. As of June 2021, MIGA has also issued 5.6 billion US Dollars of guarantees through our COVID-19 Response Program and anticipate an expansion to 10–12 billion US Dollars over the coming years, a testament to the countercyclical role that MIGA can play in mobilizing private investment in the face of the pandemic. A member of the World Bank Group, MIGA is committed to strong development impact and promoting projects that are economically, environmentally, and socially sustainable. MIGA helps investors mitigate the risks of restrictions on currency conversion and transfer, breach of contract by governments, expropriation, and war and civil disturbance, as well as offering credit enhancement on sovereign obligations.Publication Argentina Country Climate and Development Report(World Bank, Washington, DC, 2022-11)The Argentina Country Climate and Development Report (CCDR) explores opportunities and identifies trade-offs for aligning Argentina’s growth and poverty reduction policies with its commitments on, and its ability to withstand, climate change. It assesses how the country can: reduce its vulnerability to climate shocks through targeted public and private investments and adequation of social protection. The report also shows how Argentina can seize the benefits of a global decarbonization path to sustain a more robust economic growth through further development of Argentina’s potential for renewable energy, energy efficiency actions, the lithium value chain, as well as climate-smart agriculture (and land use) options. Given Argentina’s context, this CCDR focuses on win-win policies and investments, which have large co-benefits or can contribute to raising the country’s growth while helping to adapt the economy, also considering how human capital actions can accompany a just transition.