Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment
American Economic Review
vol. 112,
no. 9, September 2022
(pp. 3124-36)
See also: Original paper by by Brodeur, Cook, and Heyes (2020)
See also: Reply by Brodeur, Cook, and Heyes (2022)
See also: Reply by Brodeur, Cook, and Heyes (2022)
Abstract
Brodeur, Cook, and Heyes (2020) study hypothesis tests from economic articles and find evidence for p-hacking and publication bias, in particular for instrumental variable and difference-in-difference studies. When adjusting for rounding errors (introducing a novel method), statistical evidence for p-hacking from randomization tests and caliper tests at the 5 percent significance threshold vanishes for difference-in-difference studies but remains for instrumental variable studies. Results at the 1 percent and 10 percent significance thresholds remain largely similar. In addition, Brodeur, Cook, and Heyes derive latent distributions of z-statistics absent publication bias using two different approaches. We establish for each approach a result that challenges its applicability.Citation
Kranz, Sebastian, and Peter Pütz. 2022. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment." American Economic Review, 112 (9): 3124-36. DOI: 10.1257/aer.20210121Additional Materials
JEL Classification
- A14 Sociology of Economics
- C12 Hypothesis Testing: General
- C52 Model Evaluation, Validation, and Selection