Data and Code for: Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics

Principal Investigator(s): View help for Principal Investigator(s) Abel Brodeur, University of Ottawa; Nikolai Cook, University of Ottawa; Anthony Heyes, University of Ottawa

Version: View help for Version V2

Name	File Type	Size	Last Modified
code			02/16/2022 08:06:AM
data			02/04/2022 09:24:AM
environment			03/25/2021 11:08:AM
metadata			03/25/2021 11:08:AM
.gitignore	text/plain	7 bytes	03/25/2021 07:07:AM
Changelog.txt	text/plain	2.7 KB	02/16/2022 03:07:AM
REPRODUCING.md	text/x-web-markdown	2.1 KB	03/25/2021 07:07:AM

Project Citation:

Brodeur, Abel, Cook, Nikolai, and Heyes, Anthony. Data and Code for: Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics. Nashville, TN: American Economic Association [publisher], 2022. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2022-02-16. https://doi.org/10.3886/E120246V2

Project Description

Summary: The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that: (1) Papers published in the `Top 5' journals are different to other; (2) The journal revise and resubmit process mitigates the problem; (3) Things are improving through time.

Scope of Project

Subject Terms: p-hacking; publication bias; research methods; causal inference; p-curves

JEL Classification:
      A11 Role of Economics; Role of Economists; Market for Economists
      B41 Economic Methodology
      C13 Estimation: General
      C40 Econometric and Statistical Methods: Special Topics: General
      I23 Higher Education; Research Institutions

Time Period(s): 1/1/2015 – 12/31/2018 (2015 and 2018)

Data Type(s): other

Related Publications

Download this project

Published Versions

V2 [2022-02-16]

V1 [2020-09-15]

Export Metadata

Dublin Core

DDI 2.5

Report a Problem

Found a serious problem with the data, such as disclosure risk or copyrighted content? Let us know.

This material is distributed exactly as received from the data depositor. As of April 2026, depositors are required to submit study materials in accessible formats. ICPSR has not reviewed, checked, or processed this material. For additional information about the study, please contact the investigator(s) directly. If you have questions about the accessibility of materials distributed by ICPSR or require further assistance, please visit ICPSR's Accessibility Center.