Do journals enforce their dataset-must-be-provided policies?

Question

At lunch today, a law prof was asking if econ journals that require datasets to be provided for accepted papers actually enforce that policy. He was ready to at a journal, but they never asked him for his data.
Anybody else find that?

Answer 1

The most recent evidence on this I've seen is that journals have very mixed enforcement of their policies. See https://www.edawax.de/2018/04/new-paper-asks-do-journals-journals-enforce-their-data-policies/  and https://replicationnetwork.com/2018/10/20/vlaeminck-podkrajac-do-economics-journals-enforce-their-data-policies/ for discussions. The paper itself is here: https://iassistquarterly.com/index.php/iassist/article/view/6

They authors find that about half of the articles they looked at complied with the data policies, but glancing at their data (http://journaldata.zbw.eu/dataset/journals-in-economic-sciences-paying-lip-service-to-reproducible-research-replication-data) it seems that journals had either very high or very low compliance rates.

Here's another paper that looks at articles in Science: https://www.pnas.org/content/115/11/2584 (the quotes from researchers who are asked for code/data are the best part)

Comments

I think there might be a need/demand for reviewers just limited to verifying replicability.